Although Twitter constitutes as one of the primary sources of realtime news with users acting as the sensors updating the content from all across the globe, yet the spread of rumours via Twitter is becoming an increasingly alarming issue and is known to have caused significant damage already. We propose a credibility analysis approach based on the linguistic structure of the tweets. We not only characterize the Twitter events but also predict their perceived credibility of them by a novel deep learning architecture. We use the huge CREDBANK data to conduct our experiments. Some of our exciting findings are that standard LIWC categories like ‘negate’, ‘discrep’, ‘cogmech’, ‘swear’ and the Empath categories

like ‘hate’, ‘poor’, ‘government’, ‘worship’ and ‘swearing-terms’ correlate negatively with the credibility of events, While some of our results resonate with the earlier literature others represent novel insights of the fake and legitimate twitter events. Using the above observations and the current deep learning architecture we predict the credibility of an event (a four-class classification problem in our case) with an accuracy of 0.54 that improves the best-known state-of-the-art (current accuracy 0.43) by ∼ 26%. A fascinating observation is that even by looking at the first few tweets of an event, it is possible to make the prediction almost as accurate as in the case where the entire volume of tweets is observed.

ACM Reference Format: Jasabanta Patro and Pushpendra Singh Rathore. 2020. A Sociolinguistic Route to the Characterization and Detection of the Credibility of Events on Twitter. In Proceedings of the 31st ACM Conference on Hypertext and Social Media (HT ’20), July 13–15, 2020, Virtual Event, USA. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3372923.3404795

In the current decade, social media platforms have seen a burgeoning growth, allowing easy access to the knowledge base and enabling millions of people to connect with each other[1]. Online sites avail information on a range of topics to their users. Users, on the other hand, use social media for a variety of purposes, which include information sharing, gathering and expression of personal

1No. of users in different social media :https://www.statista.com/statistics/272014/ global-social-networks-ranked-by-number-of-users/

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. HT ’20, July 13–15, 2020, Virtual Event, USA © 2020 Association for Computing Machinery. ACM ISBN 978-1-4503-7098-1/20/07...$15.00 https://doi.org/10.1145/3372923.3404795

thoughts or opinions. For instance, in an emergency, some users generate information by unfolding observations that they have witnessed about or by bringing relevant knowledge from external sources into social media. From this pool of information, other users try to synthesize or derive elaborate interpretations continuously over time.

Even though this process can generate, gather, filter and propagate information quite rapidly, unfortunately, unlike in traditional news media, they don’t have gatekeepers who can control the quality of information flowing across [8]. This makes the social media platforms a vulnerable place, where one can quickly get susceptible to spam content [19], stealth advertising [38, 48], and the propagation of rumor [72] or misinformation [10, 31, 36, 41, 42]. For example, immediately after the 2010 earthquake in Chile, several rumours were posted and re-posted on Twitter in the absence of any official statement, which raised panic, chaos and insecurity among the residents [31]. Recently, the social computing community has started taking increased interest in understanding and mitigating such vulnerabilities. There are works that focus on automatic detection of contradictory information [64], investigating trolling [28] and spread of misinformation [10]. In order to tackle these vulnerabilities, there is a need for an automatic system that can measure the credibility or the “perceived trustworthiness” of different topics that people discuss in social media platforms. Most often these topics are referred to as social media events. These are real-world events reported over social media at specific times, i.e., each event is associated with a time-ordered stream of social media posts.

Many of these studies found that the linguistic features are good indicators of credibility, e.g., expressions of anxiety, uncertainty, etc. are considered as good signals of credibility [8, 56, 71]. While, earlier researchers have mostly focused on specific events that were instances of misinformation, such as the spread of rumors during the 2011 Great East Japan earthquake [27], the 2013 Boston marathon bombings [29], the 2014 Sydney siege event [3] etc., the recent introduction of the CREDBANK data [34] has opened up a new opportunity to study credibility at a macroscopic level. The parsimonious language model [35] defined over this dataset is the current state-of-the-art model to predict the credibility of an event. They found that the language used by millions of people on social media has considerable information about an event’s credibility [35]. Less credible social media events are more questioned and are viewed as suspicious as they lack strong evidences backing them. Post from other users in such scenario have expressions of skepticism like ‘hard to believe’, ‘but’, ‘maybe’ etc. along with anxiety in their words. On the other hand, highly credible events exhibit more positive sentiment expressions such as ‘wow’, ‘so good!!’, etc. and a general agreement among the users which can be observed

through expressions such as ‘I agree’, ‘convincing’ etc. This is since the information is usually backed by strong evidences or are from reliable sources [73].

1.1 Working definition of credibility

1.2 Operationalizing the definition of credibility

1.3 Contributions and key results

2.1 Social media: a rising reserve of information and its diffusion

2.2 Linguistic structure related to event credibility

2.3 Present work

As we have already stated, we consider the largest available dataset, i.e., the CREDBANK corpus [34] for our study. This massive dataset was constructed by iteratively tracking millions of public tweets using Twitter’s streaming API[3]. The corpus consists of over 66M tweets covering 1,377 events reported on Twitter between October 2014 and February 2015. The credibility of each event is annotated by Amazon Mechanical Turkers (AMT) which range from ‘certainly inaccurate [−2]’ to ‘certainly accurate [+2]’. To ensure that the annotations obtained are of the same standard as expert level assessments, several controlled trials were carried out before finalizing the best strategy for obtaining high-quality annotations (see Mitra et al. [35] for the details).

For the event data analysis, we investigate various linguistic characteristics associated with the tweets posted across the different events having different credibility class. Note that the 1377 events are classified into only one of the four credibility classes based on the Pca range (unlike the users posting in these events who can belong to multiple classes as discussed earlier).

3Twitter Stream API: https://developer.twitter.com/en/docs/tutorials/ consuming-streaming-data.html

(a) LIWC

(b) Empath

Table 2: Top LIWC and Empath features showing absolute high Spearman’s rank correlation with Pca across events. Each distribution is statistically significant with p < 0.001 as per the two sample t-test.

4.1 Data preprocessing

4.2 Analysis

4.3 Insights

In this section, we set our objective to predict the credibility of an event. As we have seen in the previous section, the sociolinguistic structure of the tweets posted in events of different credibility level are quite different, we leverage these features extensively in this section to improve upon the overall accuracy of the prediction models proposed by us.

5.1 Predicting the credibility of an event

We experiment with a range of machine learning models starting from simple classifiers based on linguistic features to sophisticated neural architectures that are traditionally used for text classification. Note that we restricted ourselves from trying transformer based architectures as our data size is small i.e. only 1377 number of events, and neural transformers usually need large sample size to perform effectively.

We compare the results of our experiments with the current stateof-the-art parsimonious language model [35]. The Performance measure of all experiments is calculated using stratified five-fold cross-validation. In the subsequent subsections, we describe the baselines used in our experiments, following which we describe the proposed models.

5.1.1 Baselines. Here we describe the current state-of-the-art parsimonious language model. We also describe a set of other text classification neural architectures that we use as baselines. Parsimonious language model: We consider the current state-of- the-art parsimonious language model [35] as our first baseline. Note that we have used the same dataset as is used in this baseline. The authors in this model have viewed the event credibility prediction problem as an ordered logistic regression problem. The regression model takes linguistic features computed over event tweets as input and predicts one of the ordered credibility levels as presented in Table 1 as output. The authors used three types of features – lexiconbased, non-lexicon based and control features; the list of linguistic categories in each type are shown in Table 4. Realizing that the

5https://www.kevinmd.com/blog/2012/10/physician-twitter-examining-data.html

Trend	Feature type	Feature list
decreasing credibility	LIWC	‘social’, ‘cogmech’, ‘filler’, ‘hear’, ‘i’, ‘ipron’, ‘negate’, ‘percept’, ‘conj’, ‘present’, ‘pronoun’, ‘relig’, ‘see’, ‘swear’, ‘tentat’
decreasing credibility	Empath	‘swearing-terms’, ‘hate’, ‘hearing’, ‘poor’, ‘government’, ‘worship’
increasing credibility	LIWC	‘achieve’, ‘bio’, ‘health’, ‘sad’, ‘sports’, ’discrep’
increasing credibility	Empath	‘competing’, ‘fight’, ‘fun’, ’achievement’

Table 3: List of linguistic features showing monotonically increasing and decreasing trend with respect to the credibility levels. Each distribution is statistically significant (p < 0.001, two sample t-test).

feature space is ample (over 9000) and the sample size is small (1377 events), the authors used the penalized version of logistic regression which handles multi-collinearity and sparsity. Authors have done extensive experiments to analyze the explanatory power of each feature. First control features are included in the model to understand their explanatory power, after fixing them other features are included subsequently in the model.

The regression technique has a parameter α (where 0 ≤ α ≤ 1) which determines the distribution of weights among the predictive variables. When α = 0, all correlated features are included with coefficient weights shrunk toward each other, while α = 1 includes only one representative feature per correlated cluster with other coefficients set to 0. Authors found that the model performs best at α = 1. Deep pyramid CNN (DPCNN) : We use word-level DPCNN model architecture which was earlier proposed as an efficient text categorization method [23] as a baseline. We use 15 weight layers as it has outperformed previous shallow methods across six benchmark datasets for sentiment classification and topic categorization. We keep other hyperparameters the same as in the best performing model corresponding to the previous study [23]. Character-CNN: We use char-CNN model, which was earlier pro- posed as an efficient text categorization method [70] as another baseline. This method outperformed many traditional and deep neural network model over benchmark datasets. We go for optimal combination of hyperparameters as defined in the original paper.

5.1.2 Proposed models. In this section, we describe our proposed models. As a first step, we wanted to understand whether the sociolinguistic features (LIWC/Empath) analyzed in the previous sections are at all effective for our prediction task. As a following step, we employ a modern deep learning architecture, the Hierarchical Attention Networks (HAN) for the classification of the events. The effectiveness of these models in classifying text documents [67], in predicting the edit quality of Wikipedia edits [51], and numerous other tasks where a hierarchical structure is inherited motivates the choice of the HANs. Classification of events, where every event is a collection of tweets is highly analogous to the hierarchical classification problem and hence the choice. Edit quality prediction task again can be thought of as similar to tweet quality/credibility prediction task and therefore, the choice. We also hybrid the HAN with the sociolinguistic features to harness additional benefits. Classifiers based on linguistic features: In this method, we con- sider all the sociolinguistic features described in Table 4 plus all the LIWC and Empath features. We conduct a range of experiments starting from considering individual words in various linguistic categories as features to linguistic categories themselves as features to the mixture of both the strategies. In all of our experiments, the

feature values are calculated event wise considering the tweets associated with the event. For the experiments where we consider linguistic categories as features, the features are calculated similarly as described in the event data analysis 4.2. However, in the experiments where individual words in the categories are considered as features, we consider the frequency in the event tweets as features values. Hierarchical attention network (HAN) model for event clas- sification: Yang et al. [67] introduced the HAN as a method for document classification. It has outperformed many strong baselines including linear methods [33, 70], SVMs [59], methods based on word-CNN [24], char-CNN [70], LSTM, conv-GRNN [59] and LSTM-GRNN [59]. The overall architecture of HAN is illustrated

Figure 3: The architecture of the hierarchical attention net- work (HAN).

Figure 3. In our study, we map the event classification problem as a document classification problem. We project each event as a document and each tweet in an event as a sentence in the document. The elementary construct of documents/events are the same, i.e., words. HAN has four modules (i) word encoder, (ii) word attention, (iii) tweet/sentence encoder and (iv) tweet/sentence attention. The encoder modules of HAN are GRU based sequence encoders [5]. The GRU uses a gating mechanism to track the state of sequences without using separate memory cells. There are two types of gates: the reset gate and the update gate. They together control the way information is updated in each state. The overall architecture of a HAN is illustrated in Figure 3.

For our purpose, we use negative log-likelihood of the correct labels as training loss, i.e.,

Where j is the label of the event k. We conduct all experiments using five-fold cross-validation. We use Glove vectors [44] as our input word embeddings. We keep the size of vectors as 300. We have fine-tuned the other hyperparameters to obtain the best results; the final set of hyperparameters is reported in the next section, where we demonstrate the results. Hierarchical attention network augmented with sociolinguis- tic features (HAN-SL): In this model, we augment the final event vector from HAN with the sociolinguistic features as discussed earlier at the beginning of this section. The architecture of our proposed system is shown in Figure 4: Architecture of the HAN-SL model.

Figure 4. The sociolinguistic features are first computed for each event and passed through a dense layer to get the hidden representation. Let s be the sociolinguistic vector for an event, then

hs = ReLU (Whss + bhs )

Loss = −

�

k

loдpkj

Feature type	Feature name	Lexicon size
Lexicon-bassed	Modality [14, 50, 52] Subjectivity [66] Hedges [21, 60] Evidentiality [7, 50, 54] Negation [52] Exclusions [60] Conjunction [60] Boosters [21, 60] Anxiety [60] Positive Emotion [60] Negative Emotion[60]	30 8222 125 82 12 17 28 45 91 499 408
Non-lexicon based	Number of hashtags Number of question marks Occurrence of quoted conten Number of capitalized terms	Number of hashtags Number of question marks Occurrence of quoted conten Number of capitalized terms
Control	Number of original tweets, retweets, replies Average length of original tweets, retweets, replies Number of words in original tweets, retweets, replies	Number of original tweets, retweets, replies Average length of original tweets, retweets, replies Number of words in original tweets, retweets, replies

Table 4: List of features used in the parsimonious language model [35].

The dimension size of each newly introduced dense layers is

kept at 50. The hyperparameters are reported in the next section where we discuss our results.

where, hs is the hidden representation of s. The hidden vector hs is then concatenated with the event vector v as obtained from HAN model and passed through another dense layer. Thus,

hf i = [v, hS ]

hf = ReLU (Whf hf i + bhf ) The outcome of final dense layer hf is passed through a softmax layer to get the credibility labels, i.e.,

p = sof tmax(Wchf + bc ) All weights are learned during training. We use Glove vectors

[44] as our input word embeddings; the size of vectors is kept to

5.1.3 Evaluation. The state-of-the-art baseline based on the parsimonious language model [35] introduces two different forms of evaluation – conservative and liberal. On similar lines, we use both of these techniques for the assessment. The methods are briefly described below(as described in Mitra et al. [35]). Note that other forms of measure like NDCG can be used if we recast the problem as a regression task rather than a classification task; however, since this is not possible unless we also overhaul the state-of-the-art baseline framework, we refrained from using these measures. Conservative evaluation: This evaluation strategy ignores the partial ordering present among the credibility classes. The model performance is measured based on whether the predicted credibility class label for an event exactly matches with the correct label. Liberal evaluation: This evaluation strategy rewards a partial credit of 0.5 if misclassification is done by one level and a partial credit of 0.25 if the misclassification is done by two levels. For instance, if the original class-id for an event is 1 as per the ground-truth and the model predicts the class id to be 2, then the contribution to the accuracy for this misclassification is not 0 but 0.5. Similarly, if the original class id for an event is 1 as per the ground-truth and the model predicts the class id to be 3, then the contribution to the accuracy for this misclassification is not 0 but 0.25.

Performance measurement: The calculation of accuracy is based on the confusion matrix where actual and predicted class events are mapped along with rows and columns, respectively. The accuracy is then calculated based on the degree of agreement and disagreement between the actual and predicted class of events.

k �

r =1

Therefore they were chosen[6]. For instance, LR is a well-behaved algorithm that can be trained usually if the features are roughly linear, and the problem is linearly separable. The algorithm is quite robust to noisy training data, and one can avoid overfitting. Feature selection is also possible using L2 or L1 regularization. Naïve Bayes classifiers are usually known to work well for unbalanced classes. K-neighbor algorithm is also known to be very robust to noisy training data and is more effective if the training data is astronomical. Finally, the advantage of RF/Extratree types of algorithms are that they do not expect linear features or even features that interact linearly. Another advantage is that because of how they are constructed (using bagging or boosting) these algorithms handle very well high-dimensional spaces as well as a large number of training examples. As our feature space is quite complex, RF/Extratree types of algorithms, as we shall see below, outperform other classes of algorithms.

In Table 5, we report the Acccon and Acclib values for the different feature combinations. The last entry in this table achieves the best results. We shall use this combination to refer to the ‘sociolinguistic’ features (SL) in the later parts of this paper. Note that the results for this feature combination, significantly outperform the best performing baseline model (PLM). Proposed neural models: Here we present the results obtained from the neural models. In the GRU with no attention model reported in Table 6 we considered the same HAN architecture (Figure 3) however with the attention layer removed from both the word and the tweet levels. As we can observe from Table 6, the HAN-SL model performs best in terms of both Acccon and Acclib . It is also interesting to observe the neural models are only marginally better than the linguistic feature-based models. After fine-tuning the optimal set of hyperparameters, we get for these neural models are noted in Table 7. All results reported are for this choice of hyperparameters. Class wise performance: The class-wise precision (P), recall (R) and F1-scores for the best performing baseline model, the featurebased classifier and the HAN-SL classifier are reported in Table 8. The HAN-SL classifier performs marginally better in terms of macro F1 than the linguistic feature-based model. Both the proposed models by far outperform the baseline model. Early tweets: As an exciting extension of the above experiments, we report the results of the prediction if we consider only the initial tweets were corresponding to an event. Note that this is easily doable since the tweets are ordered in a timeline. The prediction accuracies of the events considering the first 1k, 5k and the full 10k tweets per event are noted in Table 9. The significant observation is that the initial 1k tweets are almost equally informative in credibility prediction as the aggregate data.

Theoretical underpinnings: It is well-known that modern CMC systems allow users to interact using multiple modes; however, the predominant way still corresponds to textual conversations. Automatic content analysis can help in understanding various facets of human behaviour, which can immensely benefit researchers

6https://www.quora.com/What-are-the-advantages-of-different-classificationalgorithms

Accuracy =

k �

a=1

Xa,r

∗ wa,r (2) n

where xa,r is the number of events from the a[th] actual class predicted as being from r [th] class, n is the total number of events classified, wa,r is the credit for correct/incorrect classification. The credit will be assigned based on the conservative or liberal evaluation strategies as described previously. Apart from the accuracy, we also report the class-wise precision, recall and F1-score for the best performing models for a fair comparison.

5.1.4 Results and discussion. The results of the baseline and the proposed models are presented as follows: Baselines: The conservative (Acccon ) and the liberal (Acclib ) accuracy values for the different models considered as baselines in this study are shown in Table 6. From the table, it is clear that among the baselines, the parsimonious language model (PLM) outperforms all the other techniques. Proposed linguistic feature based models: The conservative (Acccon) and liberal (Acclib ) values for various classifiers over different feature constructions are shown in Table 5. We consider various feature types as follows. LIWC cat: Here individual LIWC categories are considered as fea- tures. Empath cat: Here individual Empath categories are considered as features. LIWC cat + Empath cat: Here LIWC and Empath categories jointly are considered as features. LIWC lex: Here we consider individual words in the LIWC cate- gories as features. Empath lex: Here we consider individual words in the Empath cate- gories as features. LIWC lex + Empath lex: Here we consider individual words in LIWC and Empath categories jointly as features. LIWC cat + Empath cat + OthLing Cat + Non-lexicon + Control: Here we consider all LIWC and Empath categories plus the non-lexicon, control, and other lexicon categories noted in Table 4 as features. LIWC lex + Empath lex + OthLing lex + Non-lexicon + Control: Here we consider all LIWC and Empath category words plus the other lexicon, non-lexicon, control attributes noted in Table 4 as features. LIWC cat + Empath cat + OthLing lex + Non-lexicon + Control: Here we consider all LIWC and Empath categories plus the other lexicon, non-lexicon and control attributes noted in Table 4 as features.

We have also used various types of classifiers for the classification task. In particular, we have used logistic regression (LR), random forest (RF), Extratree, K-neighbors and Naïve Bayes. We choose these classifiers for their diversity since they are known to be able to solve a vast range of different types of classification problems. Each of these classifiers represents different schools of thoughts and have their own set of strengths and advantages.

Table 5: Accuracy of different classifiers for various feature constructions.

from multiple domains. For instance, a linguist might be typically interested in investigating the correlation between language of the text and the overall commitment of the user. Also, they might be interested to find if the readability of the text is determined by the user’s perception of a topic/event. Likewise, a social scientist might be interested in how the overall credibility of a user can be inferred from the credibility of the individual topics that (s)he predominantly posts. Another point of interest could be how the common language use can help in understanding uncertainty in data. A psychologist might benefit from our study in assessing the user behaviour by observing their dominant credibility classes.

GRU with no attention 0.39 0.64 HAN (10k) 0.52 0.73 HAN-SL (10k) 0.54 0.74 Table 6: Conservative and liberal accuracy values for the baselines and the proposed neural models. The first three rows correspond to the three baselines while the last three rows corresponds to different variants of our model.

Parameter/hyperparameter Value Optimizer rmsprop Loss categoricalcrossentropy Dense layer output 50 Attention layer1 output 100 Attention layer2 output 50 GRU layer output 100 Batch size in fitting 20 Dropout none Learning rate 0.001 ρ (in rmsprop optimizer) 0.9 Table 7: Optimal set of hyperparameters for the HAN and the HAN-SL models obtained through fine-tuning.

Conservative Liberal Model Class P R F1 M-F1 P R F1 M-F1

1 0.57 0.42 0.48 0.75 0.64 0.69

PLM 23 0.380.41 0.390.56 0.380.47 0.36 0.680.66 0.680.76 0.680.71 0.64

4 0.17 0.12 0.14 0.46 0.51 0.48

Extra treeclassifier withSL features 1234 0.570.420.500.63 0.530.460.620.07 0.550.440.550.14 0.42 0.750.710.730.8 0.730.730.780.48 0.740.720.750.6 0.70

1 0.57 0.75 0.65 0.74 0.85 0.79

HAN-SL (10k) 23 0.510.54 0.350.63 0.420.58 0.45 0.750.75 0.680.77 0.710.76 0.70

4 0.5 0.09 0.15 0.75 0.45 0.56 Table 8: Class wise precision (P), recall (R) and F1 score of the best baseline and the proposed models. M-F1: Macro F1.

Model |tweets| Acccon Acclib Extra tree 10k 0.51 0.73 classifier with 5k 0.50 0.73 SL features 1k 0.49 0.72

10k 0.52 0.73

HAN 5k 0.53 0.74

1k 0.52 0.73

10k 0.54 0.74

HAN-SL 5k 0.54 0.74

1k 0.54 0.74 Table 9: Prediction of credibility of events from early tweets.

Our results further show that over the presence of ‘negation’, ‘swear’, ‘confusion’ and ‘hate’ words in an event should arouse

suspicion about the credibility of the event. In similar lines, ‘sports’, ‘play’, ‘achievement’ words seem to be correlated to the credibility

class of the users. The observations constitute the foundation stones for the design of scalable credibility assessment systems at both the event and the user level across various social media platforms.

Design issues: Computational fact checking [12] has become a trendy area of research in recent times and is known to be very beneficial in the assessment of the authenticity of a piece of information. Our methods for credibility analysis of the events can supplement these fact-checkers to no small extent. News agencies nowadays are interested in highlighting the summaries/snapshots of news items collected from social media platforms. It would be useful to automatically tag each of them with the aggregate credibility level of the posts from which the summary is generated.

One might argue that inclusion of non-textual features and, in specific, temporal features can make our predictions more accurate. However, note that one has to wait for a sufficiently long time before such longitudinal data could be accumulated. However, our method can make predictions early on time. This is possibly very important because some of the low credible events (e.g., intense hatred, entirely fake) might have the potential to wreak havoc by sparkling a communal riot or a real battle. Such forms of ‘cognitive hacking’[7] is becoming a rising concern in the online world and our work could sketch the first steps toward large scale solutions of this problem.

Our methods can also be handy in estimating levels of exaggerations in scientific and medical news items [57]. For instance, in health news, exaggerations have been widely reported that could result in serious misconceptions en masse. Such misconceptions sometimes could have very severe repercussions on the overall lifestyle of the common mass. Therefore, flagging highly exaggerated (usually low credible) news is a direct extension of our work. Since most of our framework is very generic, we believe that such extensions should be easily achievable.

In this study, we perform a detailed linguistic characterization of tweets corresponding to an event of certain credibility. We found that standard LIWC categories like ‘negate’, ‘discrep’, ‘cogmech’, ‘swear’ etc. and the Empath categories like ‘hate’, ‘deception’, ‘swearing-terms’ etc. correlate negatively with the credibility. Using

these socio-linguistic features as well as the modern deep learning machinery, we proposed models to detect the credibility of an event. We achieve an overall 26% improvement in accuracy over the best performing baseline. An exciting result that we got is even by looking at the first few tweets of an event, it is possible to make the prediction almost as accurate as in case where the entire volume of tweets is observed. Through this work we wish to show the research community that deep learning architectures augmented with sociolinguistic features are effective route to classify event credibility. Inspiring from our work one could try similar methods for various other tasks in this domain. Also one could also expriment with recent transformer based embedding to get the new state of art.

ACKNOWLEDGEMENT

[1] Sheetal D Agarwal, W Lance Bennett, Courtney N Johnson, and Shawn Walker.

A model of crowd enabled organization: Theory and methods for understanding the role of twitter in the occupy protests. International Journal of

Communication 8 (2014), 27.

[2] Nick Anstead and Ben O’Loughlin. 2014. Social media analysis and public opinion:

The 2010 UK general election. Journal of Computer-Mediated Communication 20, 2 (2014), 204–220.

[3] Ahmer Arif, Kelley Shanahan, Fang-Ju Chou, Yoanna Dosouto, Kate Starbird,

and Emma S Spiro. 2016. How information snowballs: Exploring the role of exposure in online rumor propagation. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. ACM, 466–477.

[4] Pablo Azar and Andrew W Lo. 2016. The wisdom of Twitter crowds: Predicting

stock market reactions to FOMC meetings via Twitter feeds. (2016).

[5] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural ma chine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014).

[6] J. R. Baseheart and T. C. Cox. 1993. Effects of police use of profanity on a receiver’s

perceptions of credibility. Journal of Police and Criminal Psychology 9 (1993), 9–19. Issue 2.

[7] Sabine Bergler, Monia Doandes, Christine Gerard, and René Witte. 2004. Attribu tions. Exploring Attitude and Affect in Text: Theories and Applications, Technical Report SS-04-07 (2004), 16–19.

[8] Carlos Castillo, Marcelo Mendoza, and Barbara Poblete. 2011. Information credi bility on twitter. In Proceedings of the 20th international conference on World wide web. ACM, 675–684.

[9] Hailiang Chen, Prabuddha De, Yu Jeffrey Hu, and Byoung-Hyoun Hwang. 2014.

Wisdom of crowds: The value of stock opinions transmitted through social media. The Review of Financial Studies 27, 5 (2014), 1367–1403.

[10] Xinran Chen, Sei-Ching Joanna Sin, Yin Leng Theng, and Chei Sian Lee. 2015.

Why Do Social Media Users Share Misinformation?. In JCDL.

[11] K. C. Chretien, J. Azar, and T. Kind. 2011. Physicians on Twitter. JAMA 305

(2011), 566–568. Issue 6.

[12] G. L. Ciampaglia, P Shiralkar, L. M. Rocha, J. Bollen, F. Menczer F, and A Flammini.

Computational Fact Checking from Knowledge Networks. PLoS ONE 10, 6

(2015).

[13] Evandro Cunha, Gabriel Magno, Josemar Caetano, Douglas Teixeira, and Virgilio

Almeida. 2018. Fake news as we feel it: perception and conceptualization of the term âĂIJfake newsâĂİ in the media. In International Conference on Social Informatics. Springer, 151–166.

[14] Marie-Catherine De Marneffe, Christopher D Manning, and Christopher Potts.

Did it happen? The pragmatic complexity of veridicality assessment.

Computational linguistics 38, 2 (2012), 301–333.

[15] Nicholas Diakopoulos, Munmun De Choudhury, and Mor Naaman. 2012. Finding

and assessing social media information sources in the context of journalism. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2451–2460.

[16] Ethan Fast, Binbin Chen, and Michael S Bernstein. 2016. Empath: Understanding

topic signals in large-scale text. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 4647–4657.

[17] Ethan Fast, Binbin Chen, and Michael S. Bernstein. 2016. Empath: Understanding

Topic Signals in Large-Scale Text. In CHI.

[18] Thomas D Gauthier. 2001. Detecting trends using Spearman’s rank correlation

coefficient. Environmental forensics 2, 4 (2001), 359–362.

[19] R Grover. 2011. Ad. ly: The Art of Advertising on Twitter. Businessweek, January

6 (2011).

[20] Alfred Hermida. 2012. Tweets and truth: Journalism as a discipline of collaborative

verification. Journalism Practice 6, 5-6 (2012), 659–668.

[21] Ken Hyland. 2018. Metadiscourse: Exploring interaction in writing. Bloomsbury

Publishing.

[22] Muhammad Imran, Carlos Castillo, Fernando Diaz, and Sarah Vieweg. 2015.

Processing social media messages in mass emergency: A survey. ACM Computing Surveys (CSUR) 47, 4 (2015), 67.

[23] Rie Johnson and Tong Zhang. 2017. Deep pyramid convolutional neural networks

for text categorization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vol. 1. 562–570.

[24] Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv

preprint arXiv:1408.5882 (2014).

[25] Haewoon Kwak, Changhyun Lee, Hosung Park, and Sue Moon. 2010. What is

Twitter, a social network or a news media?. In Proceedings of the 19th international conference on World wide web. AcM, 591–600.

[26] Lian Fen Lee, Amy P Hutton, and Susan Shu. 2015. The role of social media in the

capital market: Evidence from consumer product recalls. Journal of Accounting Research 53, 2 (2015), 367–404.

[27] Fang Liu, Andrew Burton-Jones, and Dongming Xu. 2014. Rumors on Social

Media in disasters: Extending Transmission to Retransmission.. In PACIS. 49.

[28] Clare Llewellyn, Laura Cram, Adrian Favero, and Robin L. Hill. 2018. Russian

Troll Hunting in a Brexit Twitter Archive. In Proceedings of the 18th ACM/IEEE on Joint Conference on Digital Libraries (JCDL ’18). 361–362.

[29] Jim Maddock, Kate Starbird, and J Haneen. 2015. Al-Hassani, Daniel E. Sandoval,

Mania Orand, Robert M. Mason, Characterizing Online Rumoring Behavior Using Multi-Dimensional Signatures. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing.

[30] Adam Marcus, Michael S Bernstein, Osama Badar, David R Karger, Samuel Mad den, and Robert C Miller. 2011. Twitinfo: aggregating and visualizing microblogs for event exploration. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 227–236.

[31] Marcelo Mendoza, Barbara Poblete, and Carlos Castillo. 2010. Twitter Under

Crisis: Can we trust what we RT?. In Proceedings of the first workshop on social media analytics. ACM, 71–79.

[32] Stuart E Middleton, Lee Middleton, and Stefano Modafferi. 2014. Real-time crisis

mapping of natural disasters using social media. IEEE Intelligent Systems 29, 2 (2014), 9–17.

[33] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.

Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems. 3111–3119.

[34] Tanushree Mitra and Eric Gilbert. 2015. CREDBANK: A Large-Scale Social Media

Corpus With Associated Credibility Annotations.. In ICWSM. 258–267.

[35] Tanushree Mitra, Graham P Wright, and Eric Gilbert. 2017. A parsimonious

language model of social media credibility across disparate events. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, 126–145.

[36] Meredith Ringel Morris, Scott Counts, Asta Roseway, Aaron Hoff, and Julia

Schwarz. 2012. Tweeting is believing?: understanding microblog credibility perceptions. In Proceedings of the ACM 2012 conference on computer supported cooperative work. ACM, 441–450.

[37] Joe Murphy, Michael W Link, Jennifer Hunter Childs, Casey Langer Tesfaye,

Elizabeth Dean, Michael Stern, Josh Pasek, Jon Cohen, Mario Callegaro, and Paul Harwood. 2014. Social media in public opinion research: executive summary of the Aapor task force on emerging technologies in public opinion research. Public Opinion Quarterly 78, 4 (2014), 788–794.

[38] Eni Mustafaraj and P Takis Metaxas. 2010. From obscurity to prominence in

minutes: Political speech and real-time search. (2010).

[39] Alexandra Olteanu, Carlos Castillo, Fernando Diaz, and Emre Kiciman. 2016.

Social data: Biases, methodological pitfalls, and ethical boundaries. (2016).

[40] Shivam B. Parikh and Pradeep K. Atrey. 2018. Media-Rich Fake News Detection: A

Survey. 2018 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR) (2018), 436–441.

[41] Jasabanta Patro, Sabyasachee Baruah, Vivek Gupta, Monojit Choudhury, Pawan

Goyal, and Animesh Mukherjee. 2018. Characterizing the spread of exaggerated news content over social media. arXiv preprint arXiv:1811.07853 (2018).

[42] Jasabanta Patro, Sabyasachee Baruah, Vivek Gupta, Monojit Choudhury, Pawan

Goyal, and Animesh Mukherjee. 2019. Characterizing the Spread of Exaggerated Health News Content over Social Media. In Proceedings of the 30th ACM Conference on Hypertext and Social Media. 279–280.

[43] James W Pennebaker, Roger J Booth, and Martha E Francis. 2007. Linguistic

inquiry and word count: LIWC [Computer software]. Austin, TX: liwc. net (2007).

[44] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe:

Global Vectors for Word Representation. In Empirical Methods in Natural Language Processing (EMNLP). 1532–1543. http://www.aclweb.org/anthology/ D14-1162

[45] Verónica Pérez-Rosas, Bennett Kleinberg, Alexandra Lefevre, and Rada Mihalcea.

Automatic Detection of Fake News. In COLING.

[46] Rob Procter, Jeremy Crump, Susanne Karstedt, Alex Voss, and Marta Cantijoch.

Reading the riots: What were the police doing on Twitter? In Policing

Cybercrime. Routledge, 5–28.

[47] James Pustejovsky, Patrick Hanks, Roser Sauri, Andrew See, Robert Gaizauskas,

Andrea Setzer, Dragomir Radev, Beth Sundheim, David Day, Lisa Ferro, et al.

The timebank corpus. In Corpus linguistics, Vol. 2003. Lancaster, UK., 40.

[48] Jacob Ratkiewicz, Michael Conover, Mark Meiss, Bruno Gonçalves, Snehal Patil,

Alessandro Flammini, and Filippo Menczer. 2011. Truthy: mapping the spread of astroturf in microblog streams. In Proceedings of the 20th international conference companion on World wide web. ACM, 249–252.

[49] Manoel Horta Ribeiro, Pedro H Calais, Yuri A Santos, Virgilio AF Almeida, and

Wagner Meira Jr. 2017. " Like Sheep Among Wolves": Characterizing Hateful Users on Twitter. arXiv preprint arXiv:1801.00317 (2017).

[50] Victoria L Rubin, Elizabeth D Liddy, and Noriko Kando. 2006. Certainty identifi cation in texts: Categorization model and manual tagging results. In Computing attitude and affect in text: Theory and applications. Springer, 61–76.

[51] S. Sarkar, B. P. Reddy, S. Sikdar, and A. Mukherjee. 2019. StRE: Self Attentive

Edit Quality Prediction in Wikipedia. In ACL.

[52] Roser Saurı. 2008. A factuality profiler for eventualities in text. Unveröf- fentlichte Dissertation, Brandeis University. Zugriff auf http://www. cs. brandeis. edu/˜ roser/pubs/sauriDiss 1 (2008).

[53] Rui Shi, Paul Messaris, and Joseph N Cappella. 2014. Effects of online comments

on smokers’ perception of antismoking public service announcements. Journal of Computer-Mediated Communication 19, 4 (2014), 975–990.

[54] Sandeep Soni, Tanushree Mitra, Eric Gilbert, and Jacob Eisenstein. 2014. Modeling

factuality judgments in social media text. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Vol. 2. 415–420.

[55] Kate Starbird, Grace Muzny, and Leysia Palen. 2012. Learning from the crowd:

collaborative filtering techniques for identifying on-the-ground Twitterers during mass disruptions. In Proceedings of 9th International Conference on Information Systems for Crisis Response and Management, ISCRAM. 1–10.

[56] Kate Starbird, Emma Spiro, Isabelle Edwards, Kaitlyn Zhou, Jim Maddock, and

Sindhuja Narasimhan. 2016. Could this be true?: I think so! Expressed uncertainty in online rumoring. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 360–371.

[57] Petroc Sumner, Solveiga Vivian-Griffiths, Jacky Boivin, Andy Williams, Christos A

Venetis, Aimée Davies, Jack Ogden, Leanne Whelan, Bethan Hughes, Bethan Dalton, et al. 2014. The association between exaggeration in health related science news and academic press releases: retrospective observational study. Bmj 349 (2014), g7015.

[58] Kang Hoon Sung and Moon J Lee. 2015. Do online comments influence the

public’s attitudes toward an organization? Effects of online comments based on individuals’ prior attitudes. The Journal of psychology 149, 4 (2015), 325–338.

[59] Duyu Tang, Bing Qin, and Ting Liu. 2015. Document modeling with gated

recurrent neural network for sentiment classification. In Proceedings of the 2015 conference on empirical methods in natural language processing. 1422–1432.

[60] Yla R Tausczik and James W Pennebaker. 2010. The psychological meaning of

words: LIWC and computerized text analysis methods. Journal of language and social psychology 29, 1 (2010), 24–54.

[61] Peter Tolmie, Rob Procter, David William Randall, Mark Rouncefield, Christian

Burger, Geraldine Wong Sak Hoi, Arkaitz Zubiaga, and Maria Liakata. 2017. Supporting the use of user generated content in journalistic practice. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 3632–3644.

[62] Daniel Trottier and Christian Fuchs. 2014. Social media, politics and the state:

Protests, revolutions, riots, crime and policing in the age of facebook, twitter and youtube. Vol. 16. Routledge.

[63] Sarah Vieweg, Amanda L Hughes, Kate Starbird, and Leysia Palen. 2010. Mi croblogging during two natural hazards events: what twitter may contribute to situational awareness. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 1079–1088.

[64] Sarah Weissman, Samet Ayhan, Joshua Bradley, and Jimmy J. Lin. 2015. Identify ing Duplicate and Contradictory Information in Wikipedia. In JCDL.

[65] Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions

of opinions and emotions in language. Language resources and evaluation 39, 2-3 (2005), 165–210.

[66] Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual

polarity in phrase-level sentiment analysis. In Proceedings of the conference on human language technology and empirical methods in natural language processing. Association for Computational Linguistics, 347–354.

[67] Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard

Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 1480–1489.

[68] Dave Yates and Scott Paquette. 2010. Emergency knowledge management and

social media technologies: A case study of the 2010 Haitian earthquake. In Proceedings of the 73rd ASIS&T Annual Meeting on Navigating Streams in an Information Ecosystem-Volume 47. American Society for Information Science, 42.

[69] Jie Yin, Andrew Lampert, Mark Cameron, Bella Robinson, and Robert Power. 2012.

Using social media to enhance emergency situation awareness. IEEE Intelligent Systems 27, 6 (2012), 52–59.

[70] Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional

networks for text classification. In Advances in neural information processing systems. 649–657.

[71] Zhe Zhao, Paul Resnick, and Qiaozhu Mei. 2015. Enquiring minds: Early detec tion of rumors in social media from enquiry posts. In Proceedings of the 24th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 1395–1405.

[72] Arkaitz Zubiaga, Ahmet Aker, Kalina Bontcheva, Maria Liakata, and Rob Procter.

Detection and resolution of rumours in social media: A survey. ACM

Computing Surveys (CSUR) 51, 2 (2018), 32.

[73] Arkaitz Zubiaga, Ahmet Aker, Kalina Bontcheva, Maria Liakata, and Rob Procter.

Detection and Resolution of Rumours in Social Media: A Survey. ACM

Comput. Surv. 51 (2018), 32:1–32:36.

[74] Arkaitz Zubiaga, Heng Ji, and Kevin Knight. 2013. Curating and contextualizing

twitter stories to assist with social newsgathering. In Proceedings of the 2013 international conference on Intelligent user interfaces. ACM, 213–224.