Relevant and timely information collected from social media during crises can be an invaluable resource for emergency management. However, extracting this information remains a challenging task, particularly when dealing with social media postings in multiple languages. This work proposes a cross-lingual method for retrieving and summarizing crisis-relevant information from social media postings. We describe a uniform way of expressing various information needs through structured queries and a way of creating summaries answering those information needs. The method is based on multilingual transformers embeddings. Queries are written in one of the languages supported by the embeddings, and the extracted sentences can be in any of the other languages supported. Abstractive summaries are created by transformers. The evaluation, done by crowdsourcing evaluators and emergency management experts, and carried out on collections extracted from Twitter during five large-scale disasters spanning ten languages, shows the flexibility of our approach. The generated summaries are regarded as more focused, structured, and coherent than existing state-of-the-art methods, and experts compare them favorably against summaries created by existing, state-of-the-art methods.

• Information systems →Information systems applications; World Wide Web; • World Wide Web →Web services.

abstractive summarization, multilingual retrieval, social media, emergency management

ACM Reference Format: Fedor Vitiugin and Carlos Castillo. 2022. Cross-Lingual Query-Based Summarization of Crisis-Related Social Media: An Abstractive Approach Using Transformers. In Proceedings of the 33rd ACM Conference on Hypertext and Social Media (HT ’22), June 28-July 1, 2022, Barcelona, Spain. ACM, New York, NY, USA, 11 pages. https://doi.org/10.1145/3511095.3531279

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. HT ’22, June 28-July 1, 2022, Barcelona, Spain © 2022 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-9233-4/22/06...$15.00 https://doi.org/10.1145/3511095.3531279

21

Carlos Castillo ICREA and Universitat Pompeu Fabra

Barcelona, Spain

chato@icrea.cat

Social media platforms such as Twitter are widely used to share information during disasters and mass convergence events [4]. During these situations, users, including eyewitnesses, media, governmental and non-profit organizations, post an enormous volume of diverse content, from personal opinions and commentary to reports and messages providing relevant information that could lead to better situational awareness. This work describes an approach to automatically summarize information posted in social media about an event, creating brief reports to help emergency response and recovery. These reports can help emergency managers better understand a developing situation and plan the following actions accordingly [7, 14].

The development of methods that automatically extract crisisrelevant information from social media has been an active line of work for many years [15]. Traditionally, crisis information extraction methods use linguistic and semantic resources mainly concentrated on one language [38]. However, there are many cases where a single crisis affects several countries or regions that speak different languages [11, 25, 45], or affects a region where the population speaks more than one language.

Previous work has shown that the information provided by social media postings is related to the language in which they are posted, and indeed messages in different languages about the same crisis often provide complementary information [26, 47]. Extracting and summarizing information from social media in only one language introduces the risk of missing valuable information. However, creating or adapting language-specific resources or methodologies for new languages is expensive and time-consuming. Therefore, current crisis informatics solutions need effective cross-lingual tools for extracting relevant information about appropriate categories of crisis-relevant information.

Our main contributions are:

• We describe a flexible, query-based, cross-lingual method for collecting from social media relevant postings in multiple languages about specific information categories. The method uses pre-trained multilingual sentence embeddings (LASER [2]) to extract postings from a general collection of crisisrelated messages. • We describe an approach for crisis summarization that takes as input relevant postings about an information category and generates a summary using a transformer-based language model (T5 [35]). We use clustering and diversification operations to create less redundant, more information-rich summaries.

We perform empirical validation using both crowdsourcing annotators and emergency management experts and release a new

<!-- source-page: 2 -->

HT ’22, June 28-July 1, 2022, Barcelona, Spain Vitiugin and Castillo

annotated dataset to evaluate multilingual crisis informatics systems.

The remainder of this paper includes a presentation of related work (§2), followed by a description of the query-based method for crisis information extraction and cross-lingual classification and summarization (§3). Next, we describe our experimental setup (§4) and the results of our analysis (§5). Finally, we present our conclusions and envisioned future work (§6).

Mining the social web for crisis-relevant information has been an active and fruitful research topic for many years. Our coverage of it focuses on overviewing methods for mining (§2.1), classification (§2.2), and summarization (§2.3) of crisis-relevant social media messages.

2.1 Mining Social Media for During Crises

Social media is a key communication channel during all kinds of crises, including natural and man-made disasters. Computational methods from many disciplines can contribute to creating mining and retrieval systems that can help emergency managers [4]. Crisis-related social spans many different categories of information, including timely messages about urgent needs from affected populations and damaged infrastructure such as bridges or roads. Together, this information is relevant for emergency response, recovery management, and assessments of the costs of damages [18] Unfortunately, most methods for mining social media during disasters described in the extensive literature on the topic are monolingual, limiting the applicability in countries using languages other than English or even in English-speaking countries with increasing multilingual urban populations [26]. The response during the disasters could be significantly improved with the ability to employ social data mining methods on user-generated data across multiple languages [47]. Cross-lingual and multilingual classification and summarization methods provide an opportunity to gather complementary information across various languages spoken in affected areas.

2.2 Classification of Crisis-Related Messages

In the recent literature on this topic, “traditional” supervised learning methods such as Naive Bayes and Support-Vector Machines (SVM) coexist with neural-network-based methods [43]. Indeed, SVM for the classification crisis-relevant social media has consistently shown to exhibit high performance, especially when combined with semantic features computed with the help of external knowledge bases [19].

Deep learning methods using various architectures have proven effective at detecting crisis-relevant messages; a popular architecture is Convolutional Neural Networks using word embeddings [25, 30]. The addition of information specific to an event type, such as hydrological information in the case of floods, has been shown to improve classification performance [8]. A particularly influential model has been Bidirectional Encoder Representations from Transformers (BERT) [9, 34], which is currently being used for various challenging Natural Language Processing (NLP) tasks, including classification. Recent papers on this topic describe end-to-end

22

transformer-based models for crisis classification tasks, demonstrating promising results [22, 24].

Language-Agnostic SEntence Representations (LASER) is an architecture to learn joint multilingual sentence representations for 93 languages. The system uses a single Bidirectional Long Short- Term Memory (BiLSTM) encoder with a shared byte-pair encoding vocabulary for all languages, coupled with an auxiliary decoder, and trained on publicly available parallel corpora. The resulting embeddings are computed using English annotated data only and transferred to any of the 93 languages without any modification [2]. LASER embeddings have also been shown to be effective in multilingual classification tasks [6, 32].

2.3 Crisis-Related Information Summarization

Social media messages are usually short and thus tend to provide fragmented information hence consolidating and summarizing information is key [39, 40]. An informative summary can help stakeholders gain situational awareness and manage critical resources effectively [48].

The main approaches used for text summarization can be categorized as either extractive or abstractive [28]. Extractive approaches construct summaries by combining selected informative phrases or sometimes whole sentences from the source text [10]. Abstractive summarization, on the other hand, generates summaries from a representation of the semantics of a given text; an abstractive summary may contain words or sentences that do not appear in the source document(s). Abstractive summarization techniques usually employ a generative approach [21, 23].

Despite the benefits of abstractive approaches, extractive approaches are still considered state-of-the-art for summarization due to their simplicity and high performance [17, 36]. However, extractive approaches often fail to include key elements useful in a report, such as answers to “what,” “who,” “where,” “when,” and “how” questions. These are important elements in the domain of

disaster and crisis management and need to be concisely incorporated into summaries [20]. Query-based approaches have been described as a helpful manner of incorporating this information to improve the quality of reports [37]. In general, abstractive methods may facilitate the generation of more informative summaries, not restricted to sentences that directly take sequences of words from the source text [29]. State-of-the-art abstractive summarization methods tend to adopt transformers and pre-trained models that have demonstrated great performance in other NLP tasks: BART [5], T5 [12], PEGASUS [41].

Our research builds upon previous work and contains two key innovations. To the best of our knowledge, (1) we are the first to describe a summarization method that retrieves crisis-relevant information using a query-based approach, and (2) we are the first to propose a transformer-based summarization model for crisisrelated messages.

In this section, we provide an overview of the proposed method, named CLiQS-CM (Cross-LIngual Query-based Summarization of Crisis Messages). An overview of the method is shown in Figure 1.

<!-- source-page: 3 -->

Cross-Lingual Query-Based Summarization of Crisis-Related Social Media HT ’22, June 28-July 1, 2022, Barcelona, Spain

First, an automatic classification model is used for detecting crisisrelevant, informative messages (§3.1). Second, cross-lingual ranking is performed on these messages (§3.2). Third, the top k ranked messages are given as input to a summarization model (§3.3).

3.1 Classification Model

A large fraction of messages posted in social media in response to a crisis event doesn’t include any informative claims beyond merely announcing that a crisis situation is developing. Hence, a key step is detecting crisis-relevant informative messages. We model this as a binary classification task and create an automated classification model that we name CLiQC-CM (Cross-LIngual Query-based Classification of Crisis Messages).

Pre-processing. Messages are preprocessed by replacing URLs and account mentions (“@user”) by specific tokens and turning hashtags into words. We preserve punctuation and stopwords, as in the next steps, we use sentence embeddings and dependency parsing.

Feature extraction. We include morphological and syntactical features. Using the Stanza Part Of Speech (POS) tagger [33], we count the number of numerals/numbers, nouns, verbs, adverbs, and adjectives in the messages. Using the Stanza dependency parser, we extract and count syntactic features indicating the presence/absence of claim-containing sentences, such as subjective nouns, compounds, roots, and modality. For both tasks, we use pre-trained models in each of the languages we work with. We normalize the count of occurrences of each type of element in a message, such as “contains N numerals/numbers,” “contains N subjective nouns,” and so on, using min-max scaling to be in [0, 1]. We also consider Named Entity Recognition (NER) features extracted using the SpaCy library [13]. These are binary features indicating whether a message contains persons’ names, the name of a place, organization, or a date. We use SpaCy’s pre-trained models for each of the languages we work with. Off-the-shelf, SpaCy supports 15 languages, including all the ones we work with, except Croatian and Tagalog. For Croatian, we use a contributed model for the Stanza package; for Tagalog, we use an open-source pre-trained model1. Finally, we include messagespecific features indicating the (min-max scaled) number of URLs and user mentions in messages.

1https://github.com/matthewgo/FilipinoStanfordPOSTagger

23

Embeddings. For representing the input data, we used sentence embeddings generated by the pre-trained transformer-based model, LASER [2]. LASER sentence-level embeddings are obtained by applying max-pooling over the output of a BiLSTM sentence encoder. The BiLSTM output is constructed by concatenating the outputs of two LSTMs working in opposite directions (forward and backward). The bidirectional encoder captures more contextual information than a single-direction Long Short Term Memory (LSTM) encoder (e.g., a left-to-right one). In our experiments, we used LASER to embed all tweet sentences into fixed-size vectors of length 1,024.

Architecture. In our classification architecture, the embeddings are passed to an LSTM-layer and then combined with additional features. This architecture is inspired by one proposed for the detection of fake news articles [3], which has also been used for emotion detection [46].

The proposed scheme, depicted in Figure 2 (minus the querybased features, which are only used by the ranking step), computes the feature vectors separately and then combines them with the help of a Multi-Layer Perceptron (MLP) layer. We use binary crossentropy as the loss function to optimize and include a soft-max layer to classify social media text into one of two classes (“crisis relevant” or “not crisis relevant”). The hyper-parameter settings of the feature extractor portions are shown in Table 1. The feature combination layer uses the softmax activation function with Adam optimizer, the learning rate of 0.001, batch size of 100, and binary cross-entropy loss.

3.2 Cross-lingual Ranking Model

The next step in our method is to retrieve, from the informative crisis-related messages, a series of messages that are relevant for various informational categories. Specifically, we retrieve and rank the top-k most relevant messages from each category to pass them to the summarization model (§3.3).

To make information extraction more adaptable to different needs of emergency managers, our method is based on structured queries. Each query is related to a specific information need and contains keywords, templates, and prototypes; a sample query is found in Table 4. Keywords are words used frequently in messages of a category; templates are fragments containing key crisis-relevant facts; and examples are entire sentences or even entire messages corresponding to each category.

Figure 1: Overview of our cross-lingual information summarization framework.

<!-- source-page: 4 -->

HT ’22, June 28-July 1, 2022, Barcelona, Spain Vitiugin and Castillo

Table 1: Values of hyper-parameters.

Each query is written in one language (English in our case) and used to extract information across all languages. Queries are, for the most part, agnostic of the type of event, but in some cases, they may include elements that are specific to a type of disaster, for instance, in the case of earthquakes, their magnitude, or in the case of storms, rainfall. We remark that context-based semantics allow our system to work even if these elements provided by the user are not 100% complete (e.g., we can find messages containing related keywords or messages with similar semantics to the examples provided but using different wording).

To calculate query similarity features, we measure average and maximum cosine similarity between the query’s keywords, templates, prototypes, and each message. As a result, we have six similarity features. For ranking messages, we use basically the same architecture as in the previous step (S3.1), with the addition of query similarity features. This is depicted in Figure 2. After removing duplicates, we pass the top 100 candidates to the summarization step. We tested with the top 20, 50, and 100 candidates, and observed that the top 100 provided the highest recall.

3.3 Summarization Model

The final component of our method creates a category-specific summary from the retrieved messages for each category. These summaries are created by T5, a pre-trained2 transformer model widely used for summarization tasks [35]. In our preliminary experiments, this model performed better than a similarly pre-trained BART-based model.

We tested two different configurations for the summarization model: a regular condition and a diversified condition. In the regular condition, we gave T5 as input the texts from the top-100 most relevant candidates and limited the length of the output text. In the diversified condition, we clustered the top-100 most relevant candidates and gave to T5 as input the texts of all the messages on each cluster, one cluster at a time; the resulting summary is the concatenation of the per-cluster summaries. For the diversification step, we first automatically find an appropriate number of clusters using the Silhouette Score. Next, we cluster texts by the K-Means method. Because we want to keep all summaries comparable in length for our experiments, we set a maximum number of clusters to four. Also, during experiments, we found that the heuristic of summarizing clusters in decreasing order by size (i.e., starting with the largest cluster) helps to generate more relevant summaries. This is akin to following the “inverted pyramid” style typically used in journalism.

2HuggingFace - https://huggingface.co

24

In this section, we describe our data collection (§4.1) and the queries used to retrieve relevant messages in various information categories (§4.2). Then, we describe the baselines for classification (§4.3) and summarization (§4.4), and the evaluation metrics used to compare the proposed method against the baselines (§4.5).

4.1 Multilingual Data Collection

Our data collection followed standard practices to collect crisisrelated social media messages from Twitter. We collected public tweets using Twitter’s public API, filtering by location-related keywords and date, without using any additional filtering (e.g., we did not restrict the query to specific languages). We considered five disaster events between January 2020 and February 2021 that received substantial news coverage internationally:

• Australian bushfires (2019-2020): period of bushfires in many parts of Australia, which, due to its unusual intensity, size, duration, and uncontrollable dimension, was considered a “megafire”;3

• Fukushima earthquake (February 2021): a 7.1 Mw or 7.3 MJMA earthquake that struck offshore east of T¯ohoku, Japan and caused significant structural damage across the T¯ohoku and Kanto regions;4

• Gloria storm (January 2020): a Mediterranean storm that affected eastern Spain and southern France with high winds and heavy rainfall;5

• Taal volcano eruption (January 2020): a phreatomagmatic eruption from its main crater that spewed ashes across Calabarzon, Metro Manila, and some parts of Central Luzon and the Ilocos Region in the Philippines, resulting in temporary closures of schools and workplaces, and disruptions of flights in the area;6

• Zagreb earthquake (March 2020): an earthquake of magnitude 5.3 Mw, 5.5 ML, which hit the capital of Croatia, causing severe damage to hundreds of buildings in its historical center.7

All messages include a “language” field computed by Twitter using a language detection model developed specifically for tweets. We counted the number of messages per language in each event. Three of the top languages were common to all of the studied events: English (ISO 639-1 code: en), Spanish (es), and French (fr).

3https://en.wikipedia.org/wiki/2019–20Australianbushfireseason 4https://en.wikipedia.org/wiki/2021Fukushimaearthquake 5https://en.wikipedia.org/wiki/StormGloria 6https://en.wikipedia.org/wiki/2020TaalVolcanoeruption 7https://en.wikipedia.org/wiki/2020Zagrebearthquake

Hyperparameter Text features LASER embeddings Similarity features

LSTM layers - 1 - MLP layers 2 3 2 MLP neurons 128;24 1024;256;128 128;24 Dropout - 0.5 - Activation relu sigmoid relu

<!-- source-page: 5 -->

Cross-Lingual Query-Based Summarization of Crisis-Related Social Media HT ’22, June 28-July 1, 2022, Barcelona, Spain

Additionally, we found several hundred messages for each event in other languages, including Catalan (ca), Tagalog (tl), Croatian (hr), German (de), Japanese (ja), Indonesian (id), and Portuguese (pt). After collecting the data, we labelled tweets or their translation to English that contained potentially informative factual information. We name this group of tweets “informative messages.” One of the authors created the ground truth by reviewing each event and hand labeling these tweets. Another author reviewed a portion of the classified tweets, and adjustments to the classification task were agreed upon when needed. Additionally, “informative messages” were reviewed by crowdworkers during the categorization task and excluded if they did not contain information related to any category. The number of annotated messages is shown in Table 2.

Next, we used crowdsourcing to further categorize the messages into various informational categories. Specifically, we employed crowdworkers through a crowdsourcing platform,8 paying the standard rate recommended by the platform. We asked three different workers to label each of the approximately 5,700 informative messages across languages. The target categories were based on an ontology from TREC-IS 2018 [27], where we grouped some lowlevel ontology categories into higher-level ones. In total, we defined nine high-level classes of information, shown in Table 3.

8SurgeHQ - https://www.surgehq.ai

25

4.2 Queries

We use a set of queries covering the nine information categories listed in Table 3. As described in §3.2, a query for an information category includes keywords, templates, and prototypes. Creating a query requires some degree of familiarity with social media messages posted during emergency situations.

Keywords are nouns and verbs usually present in messages containing a specific category of information. Practitioners could complete this task with scripts or programs to find frequent words or phrases present in previous collections of messages from past events. Templates are small fragments of text describing crisis-relevant facts. The kind of information that we seek is in situation reports or in Wikipedia disaster-related infoboxes, which are templates that Wikipedia editors use to summarize crisis information. Users can provide such templates by copy-pasting passages from these sources, replacing the numbers or locations found there with the tokens NUMBER or LOCATION. Finally, users can provide prototypes – example messages or central passages typical in category-related texts, which can be obtained by sampling diverse, informative messages from past events. We envision a specialized user interface may assist users in formulating such queries, and we plan to explore that in future work. The scope of this paper is to demonstrate the approach and provide an initial set of easily extended and refined queries. One such query is shown in Table 4.

Figure 2: Combining the transformer embeddings with morphological, syntactic, message-specific features and query similarity features using deep MLP

Table 2: Number of annotated messages for each event, including total number of messages, and number of messages labeled as informative by a human annotator. Local languages appear in bold.

lang. 1 (en) lang. 2 (es) lang. 3 (fr) language 4 language 5 dates total info total info total info total info total info

Australia bushfires 06-31.01.2020 2000 233 2000 435 2000 460 2000 285 (ja) 2000 167 (id) Fukushima earthquake 13.02.2021 2000 266 2000 529 2000 227 3000 101 (ja) 3000 153 (id) Gloria storm 17-25.01.2020 703 393 571 210 517 168 542 233 (ca) - - Taal eruption 12-17.01.2020 551 123 691 202 610 114 1500 258 (tl) 458 151 (pt) Zagreb earthquake 22-24.03.2020 537 162 509 243 520 187 1500 282 (hr) 542 163 (de)

<!-- source-page: 6 -->

HT ’22, June 28-July 1, 2022, Barcelona, Spain Vitiugin and Castillo

Table 3: Categories for multilingual information extraction, based on the ontology from TREC-IS 2018 [27]. Example messages have been paraphrased for anonymity.

4.3 Message Classification Schemes

To compare our proposed method for informative messages detection, we construct baseline models using one classical machine learning scheme (SVM) and one deep learning scheme (LSTM) that uses LASER embeddings as input features. Also, we compare our model with a cross-lingual LinearSVC-based model that uses semantic features extracted with the BabelNet knowledge base9. The complete list of proposed modeling schemes for evaluation is the following:

• LASER+SVM: this method uses pre-trained LASER embeddings; the embeddings are then classified by a Linear SVM model; • LASER+LSTM: this method uses pre-trained LASER embeddings; the embeddings are then classified by a LSTM model; • Khare [19]: this is a cross-lingual classification approach that uses additional semantic features extracted from external knowledge bases; • CrisisBERT [24]: this is an end-to-end transformer-based model for crisis classification tasks (our implementation uses the DistilBERT [42] architecture); • CLiQC-CM (ours): this is our method for classification, using a combination of LASER embeddings and tweet-related features. classified by a LSTM model.

9https://babelnet.org

26

4.4 Summarization Methods

We compare our CLiQS-CM model and its diversified variation CLiQS-D-CM against several state-of-the-art summarization models. With the exception of the LASER+LSTM+T5 method, all of the baselines use only category-related tweets as input, i.e., we simulate the best scenario in which the input is received from a perfect classifier. In our proposed models, we use the query-based model we described. The complete list of baselines for summarization that we used is the following:

• LASER+LSTM+T5: this method uses pre-trained LASER embeddings, which are passed as input to a LSTM model for category classification and then to a T5 model for summarization; • C-SKIP [36]: this is a centroid-based method using a Fast- Text skipgram model trained on the CrisisLexT26 dataset [31], improved by the use of T5 pre-trained model (originally, the method used a corpus extracted from Google News); • CXDB8 [37]: this is a queryable word-level unsupervised extractive summarizer, which is based on the text embedding framework Flair [1]. We tested this with different pretrained embeddings, including transformer-based such as BERT and XLNet; for this task and datasets, the best results were obtained with Global Vectors for Word Representation (GLOVE) embeddings; • NAFI [29]: this is an abstractive text summarization method developed specifically for crisis events;

Category Description Example message

Casualties Affected or injured people Around 150 injured people Damage Built or natural environment damage Destroying orange trees and rice paddies Danger Messages of caution or alerts RED WARNING Barcelona - Danger to life Government Official report by public agencies Local authorities continuing the search for ... Sensor Seismic activity Zargeb hit by 5.3 magnitude earthquake Service Providing a service or help Local org. provides shelter for more than 1,000 people Water Water-related messages floods in Catalonia Weather Weather updates heavy rainfall and flooding across region

Table 4: Example query. Each query includes keywords, templates, and prototypes.

Query for category: Weather

keywords: snow, weather, rain, wind, coast, mph, kmh, forecast templates: batter parts of LOC, damages from winds, pummelling the region, NUMBER km/h winds, weather forecast, bad weather, heavy snow, strong wind, storm is hitting, wind gust prototypes: Wind, rain and snow batter parts of country Storm brought around NUMBER m of snow and affect rivers Heavy rainfall, strong wind and more than NUMBER of snow across LOC Storm is hitting eastern LOC, with high winds and heavy rain Storm has battered parts of LOC and reportedly brought worth of rain Maximum gusts of wind in LOC NUMBER km / h Tonight, terrible rains in LOC Organisation has so far done NUM health care due to strong winds Gusts of wind left fallen trees

<!-- source-page: 7 -->

Cross-Lingual Query-Based Summarization of Crisis-Related Social Media HT ’22, June 28-July 1, 2022, Barcelona, Spain

• CLiQS-CM (ours): we use a combination of LASER embeddings with tweet-related features and query similarities features that are passed to a LSTM model for the ranking step and then uses a T5 model for the summarization step; • CLiQS-D-CM (ours): this is the same as CLiQS-CM but retrieves diversified (see §3.3) top-k candidates in the ranking step.

4.5 Evaluation Metrics

To evaluate the performance of the classification models, we use three standard metrics: Accuracy (ACC), Area Under the Receiver Operating Characteristic Curve (AUC), and weighted F-measure (F1). These metrics are typically used in research on social media for emergency management (e.g., [19, 25, 47]).

To evaluate the summarization models, we considered four methods. First, we annotated all summaries for factual claims and then computed, for each summary, the fraction of factual claims it contained out of the total factual claims mentioned across all summaries. Second, we computed the BERTScore [49] of each summary, which is a metric for evaluation of a text that compares them against a ground truth; in our case, an official report about the event. Third, we performed a crowdsourced evaluation of the readability of each summary across five dimensions: grammaticality, non-redundancy, referential clarity, focus, and structure and coherence [16]; five crowdsourcing workers were asked to compare summaries across each dimension. Fourth, we asked three experts in emergency management to perform a side-by-side comparison of the summaries and computed the number of times each summary was preferred.

In this section, we present the results of our evaluation and comparison with state-of-the-art methods. First, we present an evaluation of our classification method (§5.1). Next, we consider the extent to which summaries are comprehensive in terms of factual claims (§5.2). Then, we ask crowdsourcing workers to evaluate the readability of summaries (§5.3). Finally, we ask experts to perform a side-by-side comparison of the summaries (§5.5).

5.1 Cross-lingual Classification

The first experiment is a “leave-one-language-out” evaluation: for each event, the classifier is trained on data from 3 or 4 languages and tested on the last language. What we simulate here is a scenario in which we have labeled data in several languages and extract information in a new language. Table 5 shows the performance comparison of our method CLiQC-CM (Cross-LIngual Query-based Classification of Crisis Messages) with other baselines and stateof-the-art. We also perform a “leave-one-event-out” evaluation, in which we train on multilingual data for all events except one and test on the event that was left out. Results are shown on Table 6.

We can observe that in general methods based on multilingual transformers perform better than the semantic-based model by Khare. However, there are a few differences between schemes based on LASER embeddings; SVM and LSTM achieve in general the best performance, with some variations across datasets. The performance of CrisisBERT is comparable to that of the proposed method CLiQC-CM in some cases and across some metrics, but the average

27

performance of CLiQC-CM is better. We also observe that across all methods, “leave-one-event-out” seems to pose a more difficult problem than “leave-one-language-out.” This suggests that most of the multilingual methods we tested capture fairly well event-specific concepts (such as specific places, impacts, or needs) of each crisis but do not generalize so well across events.

5.2 Recall of Factual Claims

One way of measuring how informative are different summaries are, is to consider the extent to which they contain factual claims related to an information category for an event. To perform this evaluation, we manually coded each category-related factual claim in each of the generated summaries across all methods and then counted the number of claims in every summary compared to the overall claims.10 The fraction of claims contained in summary is divided by the total number of claims across all summaries if what we call the recall of factual claims. Table 7 shows the performance comparison of CLiQS-CM with other baselines and state-of-the-art. CLiQS-CM and C-SKIP outperform the other methods in both the cross-lingual and English-only evaluation, with a small advantage for CLiQS-CM.

The diversified method CLiQS-D-CM produces summaries with less factual claims than CLiQS-CM; in our observations, this is partially explained by diversification leading to more low-ranking claims to be included, and those claims are more likely to be incorrectly associated to the category under analysis. In other words, the lower we go on the list of retrieved messages for a category, the more likely we are to find messages that actually belong to other categories. As we explain in subsection 3, we create clusters from top-100 candidates, and often the number of category-related candidates is much less than 100; in this case, clusters other than the first one are likely to be non-related to a category.

5.3 BERTScore - Similarity with Official Reports

To perform this evaluation, we retrieve summaries for each event prepared by the Emergency Response Coordination Centre (ERCC).11

These summaries are created at the level of entire events and not divided by category. Hence, we create an event-level summary using each method by combining the summaries from all categories. Each event-level summary is compared against the ERCC one using BERTScore.

Table 8 shows the results of the performance comparison of CLiQS-CM against other models. Both proposed methods CLiQS- CM and CLiQS-D-CM show better performance than the baselines in all datasets except one. The exception is C-SKIP, a centroid-based extractive method, which demonstrates higher performance for one of the analyzed events (Zagreb earthquake). The BERTScore evaluation also helps us interpret the results regarding the recall of factual claims, as often the precision of CLiQS-CM is higher than the one of CLiQS-D-CM.

Additionally, the comparison of CLiQS-CM against other models, considering only English messages as input for all models, is presented in Table 9. In this monolingual evaluation, methods are closer

10These annotated summaries are part of our data release. 11ERCC Portal - https://erccportal.jrc.ec.europa.eu/

<!-- source-page: 8 -->

HT ’22, June 28-July 1, 2022, Barcelona, Spain Vitiugin and Castillo

Table 5: Results of cross-validation evaluation of message classification across languages (“leave-one-language-out”), with 4-5 languages per event: the test data contains all of the messages for an event in one language, while the training data contains messages in other language for the same event.

Table 7: Factual claims present in each summary, on average, as a percentage of the total number of factual claims across all summaries for a category and event. We consider crosslingual and English-only evaluations. The top two methods are extractive, while the remaining four are abstractive; our methods are marked with an asterisk.

Scheme Cross-lingual English

C-SKIP 28.8% 29.0% CDDB8 24.1% 17.8%

LASER+LSTM+T5 10.0% 16.0% Nafi 25.1% 28.8% CLiQS-CM 29.4% 31.2% CLiQS-D-CM 23.8% 23.5%

to each other in terms of BERTscore similarity with the reference. CLiQS-D-CM models show slightly better average performance. In the Australia bushfires dataset, C-SKIP performs better, and in the Taal volcano eruption dataset CLiQS-CM performs slightly better.

5.4 Readability evaluation

The readability of crisis reports is crucial to provide information to practitioners in an understandable way [44]. We considered five dimensions of readability: grammaticality, non-redundancy, referential clarity, focus, structure, and coherence [16]. We performed this evaluation through a crowdsourcing platform,12 and computed our results by aggregating the assessments of five different annotators. Annotators were shown an explanation of each annotation dimension before starting the evaluation.

A total of 43 evaluation rounds were performed, and in each round, the five annotators were shown independently the six summaries in random ordering. They were asked to pick one of them

12SurgeHQ - https://www.surgehq.ai/

28

as the best in terms of each evaluation dimension. Then, we computed the best method for each round by majority voting among the five annotators. The results of the evaluation, shown in Table 10, indicate that annotators considered summaries generated by CLiQS- D-CM as less redundant, more referentially clear, more focused, and more structured and coherent than the summaries generated by other methods. On the other hand, C-SKIP summaries, which are extractive summaries generated by a centroid-based method, were considered as more grammatically correct.

5.5 Expert Evaluation

The last evaluation involved three experts in emergency management, none of them a co-author of this paper, working in three different EU countries: (1) an operations coordinator with a VOST organization, (2) a program manager at a Civil Protection Department, and (3) a project manager at an Emergency Management System. Experts were shown 43 pairs of summaries randomly selected from the five events; in each pair, which one of the summaries was generated by our method and the other by one of the baselines. Each summary was accompanied by references (links) to source tweets related to each sentence or passage in the summary, as in the following example, in which underlined letters represent links:

at least 20,000 people have taken refuge in evacuation centers. evacuees need masks. there are also many evacuees in need. evacuees need food, water, shelter and medical help.[a] more than 30 thousand people evacuated due to the eruption of the Taal volcano in the Philippines. Taal volcano eruption threatens the lives of more than 900,000 inhabitants. eruption of the Taal volcano in the Philippines has caused more than 24,000 people to be evacuated. [b,c,d]

The evaluation was performed through an online form and was “blind” in the sense that the experts did not know, in each pair,

which summary was generated by which method; the ordering

Australia bushfires Fukushima earthquake Gloria storm Taal eruption Zagreb earthquake Average Schemes ACC F1 AUC ACC F1 AUC ACC F1 AUC ACC F1 AUC ACC F1 AUC ACC F1 AUC

LASER+SVM 92.3 91.1 70.7 97.4 97.3 85.4 83.6 81.6 66.6 95.4 94.4 77.9 92.9 92.9 90.6 92.3 91.5 78.2 LASER+LSTM 91.9 74.6 92.7 97.5 92.7 99.5 83.2 66.9 86.2 94.5 84.3 96.7 93.7 83.0 97.2 92.2 80.3 94.4 Khare 88.8 85.8 64.1 88.1 85.3 65.9 56.3 50.1 70.5 90.6 86.1 50.0 41.6 30.2 33.3 73.1 67.5 58.0 CrisisBERT 91.4 87.1 96.2 97.6 96.6 99.4 87.7 83.4 94.2 95.3 92.6 98.0 94.1 90.6 98.3 93.2 90.1 97.2 CLiQC-CM 95.9 93.6 99.2 97.6 96.3 99.4 93.0 90.9 97.8 95.6 93.3 98.1 93.2 88.5 98.6 95.1 92.5 98.6

Table 6: Results of cross-validation evaluation of message classification across events (“leave-one-event-out”): the test data contains all of the messages for one event, while the training data contains messages from all of the other events.

Australia bushfires Fukushima earthquake Gloria storm Taal eruption Zagreb earthquake Average Schemes ACC F1 AUC ACC F1 AUC ACC F1 AUC ACC F1 AUC ACC F1 AUC ACC F1 AUC

LASER+SVM 90.1 87.8 63.0 96.4 96.2 84.8 87.3 85.8 71.4 93.1 91.6 65.6 85.7 84.0 74.0 90.5 89.1 71.8 LASER+LSTM 87.5 66.4 87.5 96.6 83.4 99.0 80.3 63.5 80.3 93.6 81.4 96.9 84.2 75.4 92.5 88.4 74.0 91.2 Khare 86.2 83.3 54.8 93.4 93.4 81.9 80.8 77.1 57.9 90.2 88.4 58.8 80.6 77.4 65.9 86.2 83.9 63.9 CrisisBERT 88.0 88.0 93.3 96.7 95.9 99.3 83.3 82.2 91.7 94.6 93.4 98.0 87.0 82.1 94.4 89.9 88.3 95.3 CLiQC-CM 89.1 88.2 94.0 96.1 95.9 98.4 86.4 84.8 93.3 93.7 93.5 96.3 88.4 84.4 95.0 90.7 89.4 95.4

<!-- source-page: 9 -->

Cross-Lingual Query-Based Summarization of Crisis-Related Social Media HT ’22, June 28-July 1, 2022, Barcelona, Spain

Table 8: Comparison of cross-lingual summaries against reports by ERCC using BERTScore: precision (P), Recall (R), and F1 measure (F1). The top two methods are extractive, while the remaining four are abstractive; ours are marked with an asterisk.

of each pair was randomly chosen. We performed two evaluation rounds, the first one comparing CLiQS-D-CM against Nafi, and the second one comparing CLiQS-D-CM against LSTM+LASER+T5. Experts were asked to chose in a 5-points scale whether they (1) preferred summary 1, (2) had a slight preference for summary 1, (3) considered both summaries equally preferable, (4) had a slight preference for summary 2, or (5) preferred summary 2. We additionally asked respondents to optionally comment on the quality of both summaries, if considered appropriate.

The results of the evaluation, shown in Table 11, indicate that the consulted practitioners clearly preferred summaries generated by the proposed method in 64% of the cases when compared with Nafi, and in 56% of the cases when compared with LSTM+LASER+T5. Practitioners’ opinions about summaries mentioned that CLiQS-D- CM summaries were “more accurate and with fewer repetitions,” “more information and better organized,” “better explained,”“more

understandable,” and they contain “fewer mistakes and more data.” In comparison, according to their comments, they tended to reject summaries that contain “contradictory information,” “a lot of

29

repetitions,” “more mistakes and subjects mixed.” Inspecting the evaluation, we noticed that in general CLiQS-D-CM’s had lower performance in summaries related to the Zagreb earthquake event. In contrast, in other events, our method was often preferred.

We have described a method for generating informative reports about crises from multilingual social media. This method is based on structured queries, which are matched against messages that potentially contain the information we are interested on. Queries are straightforward to construct, which means this method can be extended to a large variety of information needs. Experiments with five different disaster events indicate that we can generate highquality, readable reports from the messages and that practitioners might prefer the summaries generated by CLiQS-D-CM to those generated using state-of-the-art methods.

Australia bushfires Fukushima earthquake Gloria storm Taal eruption Zagreb earthquake Average Schemes P R F1 P R F1 P R F1 P R F1 P R F1 P R F1

C-SKIP 78.9 79.5 79.2 79.9 84.0 81.9 79.5 81.5 80.5 80.6 80.7 80.7 82.5 82.8 82.6 80.3 81.7 81.0 CXDB8 75.2 77.4 76.3 78.5 80.8 79.6 76.3 78.1 77.2 77.3 78.8 78.0 77.6 78.9 78.2 77.0 78.8 77.9

LSTM+LASER+T5 79.5 78.6 79.0 81.8 83.8 82.8 79.9 81.5 80.7 75.5 78.3 76.9 82.3 81.9 82.1 79.8 80.8 80.3 Nafi 76.0 78.6 77.3 78.9 82.1 80.5 77.6 79.8 78.7 77.0 79.8 78.4 80.3 82.4 81.3 78.0 80.5 79.2 CLiQS-CM 81.0 80.6 80.8 80.6 85.4 82.9 81.2 82.0 81.6 82.7 81.2 81.9 81.8 82.2 82.0 81.5 82.3 81.9 CLiQS-D-CM 80.5 80.1 80.3 83.1 84.8 83.9 80.9 82.2 81.5 82.4 81.1 81.7 80.6 82.6 81.6 81.5 82.2 81.8

Table 9: Comparison of English-only summaries against reports by ERCC using BERTScore: precision (P), Recall (R), and F1 measure (F1). The top two methods are extractive, while the remaining four are abstractive; ours are marked with an asterisk.

Australia bushfires Fukushima earthquake Gloria storm Taal eruption Zagreb earthquake Average Schemes P R F1 P R F1 P R F1 P R F1 P R F1 P R F1

C-SKIP 78.2 79.2 78.7 79.8 80.7 80.3 77.4 78.9 78.2 77.9 80.1 79.0 80.2 81.0 80.6 78.7 80.0 79.3 CXDB8 75.8 77.2 76.5 79.9 79.2 79.5 76.1 77.1 76.6 75.7 77.6 76.7 79.2 78.9 79.0 77.3 78.0 77.7

LSTM+LASER+T5 75.9 78.1 77.0 81.9 80.8 81.3 78.0 79.1 78.5 78.0 79.7 78.9 81.4 81.3 81.3 79.0 79.8 79.4 Nafi 75.0 78.2 76.6 77.6 80.9 79.2 75.3 77.8 76.6 72.9 78.0 75.4 77.5 79.7 78.6 75.7 78.9 77.3 CLiQS-CM 77.9 78.8 78.3 80.4 80.7 80.5 77.6 79.4 78.5 77.9 80.2 79.0 80.8 82.2 81.5 78.9 80.3 79.6 CLiQS-D-CM 78.2 78.8 78.5 81.7 81.4 81.6 79.0 79.8 79.4 77.7 80.0 78.8 81.6 82.5 82.1 79.6 80.5 80.1

Table 10: Readability evaluation of summaries, expressed as the percentage of times a method was chosen as the best for a given dimension (column). The top two methods are extractive, the remaining four are abstractive; ours are marked with an asterisk.

Non- Referential Structure and Scheme Grammaticality Redundancy Clarity Focus Coherence

C-SKIP 33.5% 14.9% 14.9% 10.2% 13.0% CDDB8 0.5% 4.2% 2.8% 2.8% 1.4%

LSTM+LASER+T5 19.5% 25.6% 20.0% 23.7% 25.6% Nafi 1.4% 1.4% 5.6% 4.2% 1.4% CLiQS-CM 20.0% 20.9% 27.0% 24.7% 24.2% CLiQS-D-CM 25.1% 33.0% 29.8% 34.4% 34.4%

<!-- source-page: 10 -->

HT ’22, June 28-July 1, 2022, Barcelona, Spain Vitiugin and Castillo

Table 11: Expert evaluation results: percentage of answers received, aggregated across three experts.

Limitations. The applicability of our method depends on some resources that are multilingual but that do not cover all existing languages. Currently, we use Stanza for POS tagging, and dependency parsing [33] (50 languages available through community contributions) and LASER embeddings to represent sentences (93 languages available). Our method only works with 50 languages in the intersection of those supported by the POS tagger, dependency parser, and embeddings. Using the proposed method for unsupported languages may require training new models for the additional languages. In the work, we generated only English summaries which were useful for practitioners’ evaluation. The generation of summaries in other languages could show different results. The proposed approach is flexible and allows including additional categories of information with help of queries, but we have not tested that at this point. Finally, the use of sentence embeddings (LASER) allows using the same approach for other social media (Facebook, Reddit, etc.) but this would require additional experiments for performance evaluation.

Future work. Our structured queries (keywords, templates, prototypes) are written in English and then used to rank informative messages in other languages; it is possible that having multilingual queries could improve the performance of the system. We are also summarizing information from social media and comparing it against official reports while using both sources (e.g., performing contrastive summarization) could help to generate summaries with extended informativeness and improved veracity. Our method assumes that messages are geolocated through some external method, while we could make our method aware of the location of the messages and/or incorporate location elements into the queries. An envisioned next step for our research is to aggregate and consolidate extracted claims belonging to the same geographical region or event and present these summaries through a map and other information products, potentially integrating them with other information sources.

Reproducibility. All of the data and code used for the experiments presented in this paper will be made freely available in a public repository with the camera-ready version of this paper. Code and data are available in a repository https://github.com/vitiugin/CLiQS- CM.

Acknowledgments. This work has been partially supported by: "la Caixa" Foundation (ID 100010434), under the agreement LCF/PR/PR16/51110009; the Ministry of Science and Innovation of Spain with project "COMCRISIS", reference code PID2019-109064GB- I00; and the EU-funded "SoBigData++" project, under Grant Agreement 871042.

[1] Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embed-

dings for sequence labeling. In Proceedings of the 27th International Conference on Computational Linguistics (COLING). 1638–1649.

30

[2] Mikel Artetxe and Holger Schwenk. 2019. Massively multilingual sentence

embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics 7 (2019), 597–610. [3] Gaurav Bhatt, Aman Sharma, Shivam Sharma, Ankush Nagpal, Balasubramanian

Raman, and Ankush Mittal. 2017. On the benefit of combining neural, statistical and external features for fake news identification. arXiv preprint arXiv:1712.03935 (2017). [4] Carlos Castillo. 2016. Big crisis data: social media in disasters and time-critical

situations. Cambridge University Press. [5] Yisong Chen and Qing Song. 2021. News Text Summarization Method based

on BART-TextRank Model. In Proceedings of the IEEE 5th Advanced Information Technology, Electronic and Automation Control Conference (IAEAC), Vol. 5. IEEE, 2005–2010. [6] Zewen Chi, Li Dong, Shuming Ma, Shaohan Huang, Saksham Singhal, Xian-

Ling Mao, He-Yan Huang, Xia Song, and Furu Wei. 2021. mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP). 1671–1683. [7] Joao Porto De Albuquerque, Benjamin Herfort, Alexander Brenning, and Alexan-

der Zipf. 2015. A geographic approach for combining social media and authoritative data towards identifying useful information for disaster management. International Journal of Geographical Information Science 29, 4 (2015), 667–689. [8] Jens A de Bruijn, Hans de Moel, Albrecht H Weerts, Marleen C de Ruiter, Erkan

Basar, Dirk Eilander, and Jeroen CJH Aerts. 2020. Improving the classification of flood tweets with contextual hydrological information in a multimodal neural network. Computers & Geosciences (2020), 104485. [9] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert:

Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018). [10] Soumi Dutta, Vibhash Chandra, Kanav Mehra, Sujata Ghatak, Asit Kumar Das,

and Saptarshi Ghosh. 2019. Summarizing microblogs during emergency events: A comparison of extractive summarization algorithms. In Emerging Technologies in Data Mining and Information Security. Springer, 859–872. [11] M. J. Fuadvy and R. Ibrahim. 2019. Multilingual Sentiment Analysis on Social

Media Disaster Data. In 2019 International Conference on Electrical, Electronics and Information Engineering (ICEEIE), Vol. 6. 269–272. [12] Anushka Gupta, Diksha Chugh, Rahul Katarya, et al. 2021. Automated News

Summarization Using Transformers. arXiv preprint arXiv:2108.01064 (2021). [13] Matthew Honnibal and Ines Montani. 2017. spacy 2: Natural language understand-

ing with bloom embeddings, convolutional neural networks and incremental parsing. To appear 7, 1 (2017). [14] Qunying Huang and Yu Xiao. 2015. Geographic situational awareness: mining

tweets for disaster preparedness, emergency response, impact, and recovery. ISPRS International Journal of Geo-Information 4, 3 (2015), 1549–1568. [15] Muhammad Imran, Carlos Castillo, Fernando Diaz, and Sarah Vieweg. 2015.

Processing social media messages in mass emergency: A survey. ACM Computing Surveys (CSUR) 47, 4 (2015), 1–38. [16] Neslihan Iskender, Aleksandra Gabryszak, Tim Polzehl, Leonhard Hennig, and

Sebastian Möller. 2019. A crowdsourcing approach to evaluate the quality of query-based extractive text summaries. In Proceedings of the 11th International Conference on Quality of Multimedia Experience (QoMEX). IEEE, 1–3. [17] Aishwarya Jadhav and Vaibhav Rajan. 2018. Extractive summarization with

swap-net: Sentences and words from alternating pointer networks. In Proceedings of the 56th annual meeting of the Association for Computational Linguistics (ACL). 142–151. [18] Amir Karami, Vishal Shah, Reza Vaezi, and Amit Bansal. 2020. Twitter speaks: A

case of national disaster situational awareness. Journal of Information Science 46, 3 (2020), 313–324. [19] Prashant Khare, Grégoire Burel, Diana Maynard, and Harith Alani. 2018. Cross-

lingual classification of crisis data. In Proceedings of the International Semantic Web Conference. Springer, 617–633. [20] Jess Kropczynski, Rob Grace, Julien Coche, Shane Halse, Eric Obeysekare, Aurélie

Montarnal, Frederick Benaben, and Andrea Tapia. 2018. Identifying actionable information on social media for emergency dispatch. Proceedings of ISCRAM Asia Pacific (2018), 11. [21] Chenliang Li, Weiran Xu, Si Li, and Sheng Gao. 2018. Guiding generation for

abstractive text summarization based on key information guide network. In Proceedings of the 2018 Conference of the North American Chapter of the Association

←Prefers Prefers slightly Both equal Prefers slightly Prefers →

CLiQS-CM 63.6% 14.7% 5.4% 8.5% 7.8% Nafi CLiQS-CM 55.8% 14.0% 12.4% 9.3% 8.5% LSTM+LASER+T5

<!-- source-page: 11 -->

Cross-Lingual Query-Based Summarization of Crisis-Related Social Media HT ’22, June 28-July 1, 2022, Barcelona, Spain

for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). 55–60. [22] Hongmin Li, Doina Caragea, and Cornelia Caragea. 2021. Combining Self-training

with Deep Learning for Disaster Tweet Classification. In Proceedings of the 18th International Conference on Information Systems for Crisis Response and Management (ISCRAM). [23] Hui Lin and Vincent Ng. 2019. Abstractive summarization: A survey of the state

of the art. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 9815–9822. [24] Junhua Liu, Trisha Singhal, Lucienne TM Blessing, Kristin L Wood, and Kwan Hui

Lim. 2021. Crisisbert: a robust transformer for crisis classification and contextual crisis embedding. In Proceedings of the 32nd ACM Conference on Hypertext and Social Media (HT). 133–141. [25] V Lorini, C Castillo, F Dottori, M Kalas, D Nappo, and P Salomon. 2019. Integrating

Social Media into a Pan-European Flood Awareness System: A Multilingual Approach. In Proceedings of ISCRAM. [26] Valerio Lorini, Carlos Castillo, Steve Peterson, Paola Rufolo, Hemant Purohit,

Diego Pajarito, João Porto de Albuquerque, and Cody Buntain. 2021. Social Media for Emergency Management: Opportunities and Challenges at the Intersection of Research and Practice. In Proceedings of the 18th International Conference on Information Systems for Crisis Response and Management (ISCRAM). 772–777. [27] R McCreadie, C Buntain, and I Soboroff. 2019. TREC Incident Streams: Finding

Actionable Information on Social Media. In Proceedings of ISCRAM. [28] Nikita Munot and Sharvari S Govilkar. 2014. Comparative study of text sum-

marization methods. International Journal of Computer Applications 102, 12 (2014). [29] Nasik Muhammad Nafi, Avishek Bose, Sarthak Khanal, Doina Caragea, and

William H Hsu. 2020. Abstractive Text Summarization of Disaster-Related Document. In Proceedings of the 17th International Conference on Information Systems for Crisis Response and Management (ISCRAM). [30] Dat Tien Nguyen, Kamela Ali Al Mannai, Shafiq Joty, Hassan Sajjad, Muhammad

Imran, and Prasenjit Mitra. 2017. Robust classification of crisis-related data on social networks using convolutional neural networks. In Proceedings of the 11th International AAAI Conference on Web and Social Media (ICWSM). [31] Alexandra Olteanu, Carlos Castillo, Fernando Diaz, and Sarah Vieweg. 2014.

Crisislex: A lexicon for collecting and filtering microblogged communications in crises. In Proceedings of the 8th International AAAI Conference on Weblogs and Social Media (ICWSM). [32] Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How Multilingual is Multi-

lingual BERT?. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL). 4996–5001. [33] Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Man-

ning. 2020. Stanza: A Python Natural Language Processing Toolkit for Many Human Languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL). [34] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Im-

proving language understanding by generative pre-training. [35] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang,

Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research 21 (2020), 1–67. [36] Gaetano Rossiello, Pierpaolo Basile, and Giovanni Semeraro. 2017. Centroid-

based text summarization through compositionality of word embeddings. In Proceedings of the MultiLing 2017 Workshop on Summarization and Summary Evaluation Across Source Types and Genres. 12–21. [37] Allen Roush. 2020. CX DB8: A queryable extractive summarizer and semantic

search engine. arXiv preprint arXiv:2012.03942 (2020). [38] Koustav Rudra, Subham Ghosh, Niloy Ganguly, Pawan Goyal, and Saptarshi

Ghosh. 2015. Extracting situational information from microblogs during disaster events: a classification-summarization approach. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management (CIKM). 583–592. [39] Koustav Rudra, Pawan Goyal, Niloy Ganguly, Prasenjit Mitra, and Muhammad

Imran. 2018. Identifying sub-events and summarizing disaster-related information from microblogs. In Proceedings of the 41st International Conference on Research anb Development in Information Retrieval (SIGIR). 265–274. [40] Koustav Rudra, Ashish Sharma, Niloy Ganguly, and Muhammad Imran. 2018.

Classifying and summarizing information from microblogs during epidemics. Information Systems Frontiers 20, 5 (2018), 933–948. [41] Ankit Sahu and Sriram G Sanjeevi. 2021. Better Fine-Tuning with Extracted

Important Sentences for Abstractive Summarization. In Proceedings of the International Conference on Communication, Control and Information Sciences (ICCISc), Vol. 1. IEEE, 1–6. [42] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Dis-

tilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 (2019). [43] Anita Saroj and Sukomal Pal. 2020. Use of social media in crisis management: A

survey. International Journal of Disaster Risk Reduction 48 (2020), 101584.

31

[44] Irina Temnikova, Sarah Vieweg, and Carlos Castillo. 2015. The case for readability

of crisis communications in social media. In Proceedings of the 24th International Conference on World Wide Web (WWW). 1245–1250. [45] Randy Joy M Ventayen. 2017. Multilingual Detection and Mapping of Emer-

gency and Disaster-related tweets. MATTER: International Journal of Science and Technology 3, 2 (2017). [46] Fedor Vitiugin and Giorgio Barnabo. 2021. Emotion Detection for Spanish by

Combining LASER Embeddings, Topic Information, and Offense Features. In Proc. of EmoEvalEs at IberLEF. [47] F Vitiugin and C Castillo. 2019. Comparison of Social Media in English and

Russian During Emergencies and Mass Convergence Events. In Proceedings of ISCRAM. [48] Himanshu Zade, Kushal Shah, Vaibhavi Rangarajan, Priyanka Kshirsagar, Muham-

mad Imran, and Kate Starbird. 2018. From situational awareness to actionability: Towards improving the utility of social media data for crisis response. Proceedings of the ACM Conference on Human-Computer Interaction (CHI) 2, CSCW (2018), 1–18. [49] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi.

BERTScore: Evaluating Text Generation with BERT. In Proceedings of the International Conference on Learning Representations.

The following complete page renders preserve figures, equations, tables, captions, footnotes, and other visual regions from the ACM version of record.