Source and attribution. Originally published by ACM in Proceedings of the 32nd ACM Conference on Hypertext and Social Media (HT ’21). DOI: 10.1145/3465336.3475119. Full text imported from the authorized ACM proceedings PDF; the terminal Visual-Meta wrapper is excluded.
<!-- Source PDF page 1; coordinate-aware two-column reading order. -->
Full Paper HT ’21, August 30–September 2, 2021, Virtual Event, Ireland
Apoorva Singh apoorva1921cs19@iitp.ac.in Indian Institute of Technology Patna Patna, India
Tanmay Sen Ericsson Kolkata, India
Sriparna Saha Indian Institute of Technology Patna Patna, India
Mohammed Hasanuzzaman ADAPT Centre, Dublin City University Dublin, Ireland
ABSTRACT
[25]. In linguistics research, the term complaining implies a conflict between expectation and reality that is the consumer expressing the complaint is infuriated or deceived by the organization or product [25]. Identification of complaint texts in natural language is crucial for:
Complaining is a speech act that is often used by consumers to signify a breach of expectation, i.e., an expression of displeasure on a consumer’s behalf towards an organization, product, or event. Complaint identification has been previously analyzed based on extensive feature engineering in centralized settings, disregarding the non-identically independently distributed (non-IID), security and privacy-preserving characteristics of complaints that can ham- per data accumulation, distribution, and learning. In this work, we propose a Bidirectional Encoder Representations from Trans- formers (BERT) based multi-task framework that aims to learn two closely related tasks, viz. complaint identification (primary task) and sentiment classification (auxiliary tasks) concurrently under federated-learning settings. Extensive evaluation on two real-world datasets show that our proposed framework surpasses the baselines and state-of-the-art framework results by a significant margin.
• linguists to gain a better understanding of the specific con- text, purpose [39]; • companies and their customer support to boost their brand- value by identifying and tending to customer-related con- cerns, thus promoting their commercial value [10]; • developers of natural language processing (NLP) applica- tions, such as dialogue systems or chatbots [21]
Formerly, sentiment analysis research has been centralized around predicting positive, negative and neutral polarities. Whereas com- plaints were thought of as minor deviation from sentiment analysis and often termed as "sentence with negative undertone with supple- mental information" [39]. Consider the following sentences about the Samsung customer support: 1. @SamsungIndia your customer support guys take forever to pick the call and assist! 2. @SamsungIndia your customer support sucks! In the traditional sense, both sentences express a negative senti- ment. However, sentence (1) expresses a complaint and also shows negative sentiment about Samsung customer support. Sentence (2) shows negative sentiment, but there is not a justifiable sign to denote any breach of expectation. Multi-task learning has frequently proven to be decisive when working with closely related tasks. Additionally, it improves the generalization process by making use of task-specific knowledge embodied in the related tasks [7]. Complaint and sentiment classi- fication are two implicitly related tasks because when an end-user expresses a grievance there is an inherent negative sentiment con- nected to it. Generally, after a negative disparity between reality and expectation no one remains upbeat. To examine this further, we learn the tasks of complaint identification (primary task) and sen- timent classification (auxiliary task) in a multi-task environment. Such methodology allows improving the prevalent single-task sys- tems for the aforesaid tasks where the feasibility of the data is limited. Federated learning [23] can serve as a connection between various companies carrying customer-specific data, by jointly build- ing pervasive models facilitating better prediction capabilities and also lays the ground for adaptive learning as per customer be- havior [5]. Furthermore, federated learning assists in shattering
CCS CONCEPTS
• Information systems →Information extraction; Expert sys- tems; Clustering and classification; Sentiment analysis.
KEYWORDS
Complaint Identification, Deep Multitask learning, Federated Learn- ing
ACM Reference Format: Apoorva Singh, Tanmay Sen, Sriparna Saha, and Mohammed Hasanuzzaman. 2021. Federated Multi-task Learning for Complaint Identification from Social Media Data. In Proceedings of the 32nd ACM Conference on Hypertext and Social Media (HT ’21), August 30-September 2, 2021, Virtual Event, Ireland. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3465336.3475119
1 INTRODUCTION
Complaining is defined as a statement of disapproval on the end user’s behalf directed towards an organization, product, or event
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. HT ’21, August 30-September 2, 2021, Virtual Event, Ireland © 2021 Association for Computing Machinery. ACM ISBN 978-1-4503-8551-0/21/08...$15.00 https://doi.org/10.1145/3465336.3475119
201
<!-- Source PDF page 2; coordinate-aware two-column reading order. -->
Full Paper HT ’21, August 30–September 2, 2021, Virtual Event, Ireland
data-related boundaries between organizations by making a larger set of data accessible to them. This is extremely crucial where the identity of the end-user or their financial information is involved [13]. Since federated learning does not require the acquisition or storage of user data, rather only the weights and gradients of the model parameters are shared with the server. It paves the path for cooperative model training without affecting data privacy and security. As this area of research has developed recently, it has been employed in multiple different scenarios. Training a classification model on actual customer grievances data originating from different organizations requires a large data sample that is an embodiment of the text that customers have trusted the organization with to be shared publicly. Moreover, to sample these complaint texts received by the organization, requires infrastructure, exclusive storage on a server, cataloging, and secu- rity guarantees. Even with rigorous access controls and security guarantees, the companies might be uncomfortable with the accu- mulation of their customer support data [18]. In the current work, we illustrate that federated learning provides a better substitute to the server-based data accumulation and training standard in a com- mercial setting. We train a Bidirectional Encoder Representations from Transformers (BERT)-based multi-task model in the federated environment for complaint identification task. The experiments are performed on two benchmark datasets Complaints dataset1 [29] and Product Review dataset2 [35]. We perform experiments maintaining the heterogeneous nature of complaint data which is in sync with practical scenarios. The main contributions of our proposed work are outlined as follows:
• To the best of our knowledge, this is the first work that in- troduces a BERT-based multi-task mechanism for combined complaint and sentiment analysis in federated Learning set- ting; • We empirically show that complaint identification is benefi- cial in a federated learning environment where the privacy, locality of data, and heterogeneity of the consumer’s data are preserved. The multi-task framework utilizes the addi- tional knowledge of sentiment analysis and outperforms the single-task framework of complaint identification. • Evaluations of the impact of the skewed datasets, number of communication rounds, and multi-task framework on the performance are studied.
The remaining parts of the paper are organized as follows: In Section 2 we review some of the previous studies in this domain. This is followed by description of the dataset and the sentiment labeling in Section 3. We discuss our proposed methodology for the federated multi-task experiments in Section 4. Next, we discuss the experimental setup and result analysis in Section 5. Finally, we conclude the paper in Section 6.
2 RELATED WORK
Complaining is a discourse act adopted by humans to express a disparity between expectation and reality. Complaints co-occur with other discourse acts such as allegation, objection, advice, or
1https://github.com/danielpreotiuc/complaints-social-media 2https://github.com/MrRaghav/Complaints-mining-from-Hindi-product- reviews/blob/master/amazon-youtube-hindi-complaints-data.xlsx
recommendation [25]. Complaints are broadly categorized as either directly aimed at the party responsible for the breach of trust (direct grievance) or indirect involvement of the party. In recent times, so- cial media has become a ubiquitous platform to express displeasure towards an organization, product, or event [29]. Consequently, this influences the brand image of the organization and directs them to address the issue immediately. Identifying complaints on social media involves detecting complaints through short text snippets and usage of implicit and sarcastic expressions, exhortations, alle- gations [26] makes it a difficult task. Prior studies have focused on the binary classification of complaints on social media in different domains [10, 29, 34, 35]. Additionally, some studies on complaints directed to public administrations have been classified based on the responsible branch [38] as well as on the escalation and feedback probability [42]. Understanding the roles of emotions and sentiments in human disposition can contribute towards building superior, efficient, and adaptable artificial intelligence (AI)-based systems [20]. Further- more, sentiment analysis has proved to be beneficial in various applications where emotion and sentiment play important roles for instance review collecting websites, recommendation system, customer support, and spam detection. Joint learning of tasks with reasonable correlation often gains from simultaneous learning. Re- cently, multi-task learning configurations have evolved into robust architectures for solving various NLP tasks [2, 30–33]. The possible justifications for this accomplishment are:
• the efficient use of the supplementary training data, • multi-task learning in some cases functions as a regular- ization process that prevents over-fitting by preserving the performance across individual tasks [6].
Federated learning design principle depends on whether the training algorithms are asynchronous or synchronous [41]. Syn- chronous large-batch training algorithms have recently garnered attention even in the data center [14]. Furthermore, in the federated learning approach, studies regarding improving privacy aspects such as secure aggregation [5] or even differential privacy [24] require synchronization on the fixed number of devices so that the server utilizes only the cumulative update from all the users involved in training. The Federated Averaging algorithm (FedAvg) [23] also works on the same principle. Even the type of sampling of the training data has a significant role in federated learning. The independent and identically distributed (IID) data usage is impracti- cal in real-life scenarios as it cannot be expected that the local data situated on the end-user devices are always IID [16]. To address this issue, Smith et al. [36] introduced a multi-task system and established MOCHA: Federated Multi-Tasks Learning standard to address the not independent and identically distributed (non-IID) data challenges. McMahan et al. [23] have illustrated that FedAvg can handle certain non-IID data. Previous studies on complaint identification have used super- vised machine learning models with a broad range of syntactic and semantic linguistic features drawn out from text in a centralized learning framework. Customizing a state-of-the-art pre-trained neural language model based on transformer networks [40] such as BERT [11] in federated learning settings is yet to be analyzed.
202
<!-- Source PDF page 3; coordinate-aware two-column reading order. -->
Full Paper HT ’21, August 30–September 2, 2021, Virtual Event, Ireland
Considering, the distinctiveness of complaint texts, the confiden- tiality aspect of the data, and the lack of existing techniques in identifying complaints, we propose a deep multi-task framework that is strengthened by the usage of a pre-trained language model for enhancing the learning process in the distributed setting. We have drawn performance comparisons between the frameworks developed for single-task and multi-task systems in the centralized as well as the federated settings. We have also collated the per- formance of the proposed federated BERT-based multi-task model with several well-known neural network models. On comprehen- sive evaluation, we observe that our proposed approach has out- performed several of the baselines by a significant margin.
3 DATASET 3.1 Dataset Description
Complaints dataset: We evaluate the model on the Complaints dataset published in the work [29]. It includes 1,232 samples of complaints class and 739 samples of non-complaints class. The dataset is further categorized in different domains based on various industry types and areas of activity, to validate analyzing complaints by domain, as shown in Table 1. To maintain a balanced set of tweets, 1478 tweets were additionally sampled consisting of two groups of 739 tweets each. The first group constitutes of ’random tweets’ that were addressed to a particular Twitter handle, and the second group consists of ’random reply’ tweets. We use the original set of 1971 tweets for our experiments as the random tweets and random replies do not belong to any particular domain. The data distribution is shown in Table 1.
Table 1: Data distribution in the Complaints-Data.
Domain Complaints Non-Complaints
Apparel 117 141 Cars 67 25 Electronics 174 112 Foods 95 35 Retail 124 75 Services 207 130 Software 189 103 Transport 139 109 Others 96 33
Total 1232 739
Product Review dataset: We have also used a low resource Hindi language-based product review dataset3 introduced in [35]. This dataset is a collection of product reviews posted on the retail web- site, Amazon India4 and the video-sharing platform, YouTube5
comment section. We report the statistics of the dataset in Table 2. The reviews have been sub-divided based on different domains or product categories as shown in Table 2.
3https://github.com/MrRaghav/Complaints-mining-from-Hindi-product- reviews/blob/master/amazon-youtube-hindi-complaints-data.xlsx 4https://www.amazon.in/ 5https://www.YouTube.com/
Table 2: Data distribution in the Product Review dataset
Domain Complaints Non-Complaints
Book 117 2772 Phone 370 365 Electronics 57 21 Miscellaneous 22 37
Total 566 3145
3.2 Weak Supervision-based Sentiment Labeling
Existing approaches for sentiment annotation are mostly based on supervised learning. As human annotation is not only tedious and time-consuming but also a very costly task, we leverage a weak supervision-based labeling approach. We generate the sentiment labels using pre-trained classifiers and not by manual tagging, so the name "weak supervision". The labels generated by this method will constitute some noisy labels as the instances are not manually annotated. However, the cost sustained due to this is nominal using such a method. The three sentiment classifiers, namely VADER6
[17], TextBlob7, and NLTK8, are utilized to generate sentiment la- bels for each of the instances in the dataset. We observed that there were some instances where TextBlob and NLTK gave better predic- tions than VADER but in the majority of the evaluations, VADER outperformed the other two classifiers by a significant margin. Out of the 500 instances that we evaluated manually, VADER predicted 427 instances correctly whereas NLTK and TextBlob predicted 415 and 398 instances, respectively. Hence, we use VADER classifier sentiment tags as sentiment labels for further experiments. The col- lection of data over the sentiment classes is 597 tweets in negative class, 607 in neutral class, and 767 in positive class. However, in the case of Product Review dataset, Hindi being one of the low-resource languages, with negligible language processing work done on it, it lacks a standard Hindi sentiment analyzer to create the sentiment labels. The reviews were first translated into English using Google Translate9, this facilitated us to use VADER to identify the sentiment of the translated reviews. Around 500 instances of the dataset were evaluated manually and 92% of them were found to be correctly translated. The dataset distribution over the 3 sentiment classes is 33 reviews in negative class, 3390 in neutral class, and 288 in positive class.
4 PROPOSED METHODOLOGY
In this section, we formulate our problem and discuss the details of the BERT-based multi-task framework for complaint identification in federated learning. The overall flow of the federated learning framework is shown in Figure 1.
6https://github.com/cjhutto/vaderSentiment 7https://textblob.readthedocs.io/en/dev/ 8https://www.nltk.org/api/nltk.sentiment.html 9https://pypi.org/project/google-trans-new/
203
<!-- Source PDF page 4; coordinate-aware two-column reading order. -->
Full Paper HT ’21, August 30–September 2, 2021, Virtual Event, Ireland
Figure 1: Flowchart of the Federated Learning framework.
4.1 Problem Statement
We formulate our problem definition as follows: Let I = < 𝑎t,𝑠t,𝑐t >𝑇 𝑡=1 be a set of T tweets where 𝑠t and 𝑐t represent the corresponding sentiment and complaint labels for 𝑎tth tweet, respectively. Here, 𝑎t 𝜖𝑇, 𝑠t 𝜖𝑆(sentiment classes) and 𝑐t 𝜖𝐶(com- plaint classes). The objective of our multi-task learning framework is to maximize the following function 𝑓that maps each instance 𝑎t to its fitting sentiment label 𝑠t and complaint label 𝑐t. For m clients, each client w 𝜖[m] possesses local data Iw and aims to maximize the following equation:
𝑎𝑟𝑔𝑚𝑎𝑥 𝜃 (Π𝑡 𝑖=0𝑃(𝑠𝑖,𝑐𝑖|𝑎𝑖;𝜃)), (1)
where 𝑎𝑖is the input sentence whose complaint label (𝑐𝑖) and senti- ment label (𝑠𝑖) are to be predicted. 𝜃denotes the model’s parameters we want to optimize.
4.2 Federated BERT-based Multi-task Learning (Fed-BMTL)
Fine-tuning with Bidirectional Encoder Representations from Trans- formers (BERT) models: BERT’s model framework is a multi-layer bidirectional Transformer encoder [11] and is established on the original work illustrated in [40]. BERT-Base encoder is made up of a stack of N = 12 identical layers with two sub-layers in each of them. The first sub-layer is a multi-head self-attention system and the second sub-layer is a position-wise fully connected feedforward network. The multi-head self-attention is defined as :
𝑀𝑢𝑙𝑡𝑖𝐻𝑒𝑎𝑑(𝑄, 𝐾,𝑉) = 𝐶𝑜𝑛𝑐𝑎𝑡(ℎ𝑒𝑎𝑑1, ...,ℎ𝑒𝑎𝑑ℎ)𝑊𝑂, (2)
where Q is the matrix of queries, K is the matrix of keys, V is the matrix of values, 𝑊𝑂is the weight matrix and trained jointly with
Figure 2: BERT-based Multi-task Model Architecture
the model and
ℎ𝑒𝑎𝑑i = 𝐴𝑡𝑡𝑒𝑛𝑡𝑖𝑜𝑛(𝑄𝑊𝑄 𝑖, 𝐾𝑊𝐾 𝑖,𝑉𝑊𝑉 𝑖). (3)
With respect to multi-head self-attention, we also define the scaled dot product attention. It is defined as follows:
𝐴𝑡𝑡𝑒𝑛𝑡𝑖𝑜𝑛(𝑄, 𝐾,𝑉) = 𝑠𝑜𝑓𝑡𝑚𝑎𝑥(𝑄𝐾𝑇
p
𝑑𝑘 )𝑉. (4)
where dk is the dimension of the Q and K matrices. We utilize the ’bert-base-uncased’ [11] pre-trained model (Un- cased: 12-layer, 768-hidden, 12-heads, 110 M parameters) to produce a sequence of tokens. The sequence of tokens are represented as follows:
204
<!-- Source PDF page 5; coordinate-aware two-column reading order. -->
Full Paper HT ’21, August 30–September 2, 2021, Virtual Event, Ireland
[𝐶𝐿𝑆],𝑤1,𝑤2,𝑤3, ...,𝑤N, [𝑆𝐸𝑃]
where the first token of the sequence is always [𝐶𝐿𝑆] which signi- fies the starting of the sequence, the token [𝑆𝐸𝑃] is used to separate a pair of sentences from each other and also signifies the end of a sentence and 𝑤is represent tokens of the sequence. Every token passes through the Token Embedding layer, Sentence Embedding layer, and the Position Embedding layer of the BERT. Subsequently, the vectors produced from these three embedding layers 𝐸𝑖are con- catenated and passed as input to the BERT encoder. We depict the final hidden vector for the𝑖-th input token as𝑇𝑖. The representation of [𝐶𝐿𝑆] forms the sequence representation 𝑉𝑖. The BERT-based multi-task architecture is shown in Figure 2.
For the Product Review dataset, we use Multilingual BERT10 (m- BERT). m-BERT is a bi-directional language model pre-trained on 104 different languages (languages which have the most number of articles on Wikipedia ). m-BERT utilizes a similar pre-training ob- jective and next sentence prediction objectives as BERT [11]. Even though it is not specifically trained with cross-lingual objective or regulated data, m-BERT is cross-lingual. It fine-tuned on super- vised data in high resource language like English and then tested on the target languages. We employ the ’bert-base-multilingual-cased’ pre-trained model (Cased: 12-layer, 768-hidden, 12-heads, 110 M parameters) to produce a sequence of tokens.
Federated Data preparation: We prepare the Complaints-Data to suit the requirement for the federated learning setting. For the distributed settings, the data is divided into 9 different domains and each domain is allotted to a distinct client. The different domains are as shown in Table1. In the case of Product Review dataset, the review instances are divided into 4 different domains that are, Books, Phone, Electronics, and Miscellaneous. Thereafter, each domain is assigned to a different client. This division of data domain-wise closely imitates practical sce- narios where the various companies jointly working may belong to different departments or industries. Subsequently, we have split the domain-wise data into training (90%) and test (10%) sets main- taining the same ratio of the complaint and non-complaint tweets in the test set as in the train set.
Federated Training: In our work, the domain-wise datasets share a common feature space in the sense that they are all textual data. However, they hold distinct samples in comparison to each other based on the variety of domains. All of the selected clients pitch in their portion of data to train the global complaint identification model. The training process is as follows:
• Step I: Initially, the server trains a global model, the trained weights of the model are shared with the local clients. • Step II:The local participating clients fit the global model on their local training data by minimizing the loss function, for a given number of epochs. • Step III: The difference between the local client’s final model and the global model (gradient) is shared by the respective client to the server.
10https://github.com/google-research/bert/blob/master/multilingual.md
• Step IV: After accepting all the client gradients, the server combines the gradients and performs an ensemble of the weights of a subset of the local models. This is a necessary step because local models may vary quality-wise (due to the imbalance distribution of data per domain as depicted in Table 1 and 2). • Step V: The server then sends the updated weights to the clients which are then used by each client to update their local model with the global gradients. Steps II to V is repeated for a given number of communication rounds. The pseudo-code of federated training is shown in Algo- rithm 1.
Algorithm 1: Federated Average
1 Suppose, the length of the set of unique domains (or local server), D = {𝐷1, 𝐷2, . . . , 𝐷𝑁} is 𝑁.
2 Initialize, global weight as 𝑤𝑔𝑙𝑜𝑏𝑎𝑙and 𝐺𝑙𝑜𝑏𝑎𝑙𝐸𝑝𝑜𝑐ℎ𝑠(= 𝐾0)
3 for 𝑖= 1, 2, · · · , 𝐾0 do
4 Randomly select 𝑛= ⌈𝑁× 0.7⌉, number of domains (buckets) from D, sample without replacement, say, domains = {𝐷1, 𝐷2, . . . , 𝐷𝑛}
5 listOfLocalModel = list()
6 for 𝑛0 = 1,2,...,n (number of local servers) do
7 𝑤𝑏= localModelWeights(𝑤𝑔𝑙𝑜𝑏𝑎𝑙)
8 listOfLocalModel.append(𝑤𝑏)
9 end
10 FedAvg = 1
𝑛×sum(listOfLocalModel)
11 𝑤𝑔𝑙𝑜𝑏𝑎𝑙= FedAvg
12 end
13 def localModelWeights(w):
14 𝑤0 = 𝑤
15 for e = 0,1, ...,LocalEpochs do
16 𝑤𝑒+1 = 𝑤𝑒-𝜆▽Multi-task Loss(𝑦, ˆ𝑦)
17 Stop if |𝑤𝑒+1 −𝑤𝑒| < tolerance
18 end
19 return 𝑤𝑒+1
BERT-based Multi-task model: Figure 2 shows an end-to-end BERT-based multi-task model that accepts tweet (Input Sentence) as an input and generates two outputs: Complaint/Non-complaint label and Sentiment class. The model comprises a pre-trained BERT encoder which generates the representation of the input sentence and outputs the vector through the CLS token. We tweak the NSP- Dense layer of the pre-trained BERT model by taking its output 𝑉𝑖
and passing through two channels of task-specific dense layers and respective output layers (exclusive to each task). Each task-specific channel consists of two successive dense layers of 100 neurons each. Model quality is evaluated using Accuracy and macro-F1 metrics on the federated domain-wise test data.
Calculation of loss: We take Categorical Crossentropy into consid- eration as the cost function (𝐿CE) for the complaint and sentiment tasks. The loss function is as follows:
𝑃 Õ
𝑄 Õ
𝐿CE(𝑎, 𝑣) = −1
𝑖=0 (𝑡ij ∗𝑙𝑜𝑔(𝑣ij)). (5)
𝑃
𝑗=0
205
<!-- Source PDF page 6; coordinate-aware two-column reading order. -->
Full Paper HT ’21, August 30–September 2, 2021, Virtual Event, Ireland
a is the actual value and v is the predicted value. The double sum- mations from j = 0 to P and i = 0 to Q are over the number of tweets (P) and the number of classes (Q), respectively. The neural network produces a vector of Q probabilities illustrating the likelihood of a tweet being associated to the particular class. The final loss func- tion involving two categorical cross-entropy losses is described as follows: 𝐿= 𝑥∗𝐿CCE + 𝑦∗𝐿SCE. (6)
To obtain the overall loss, we compute the weighted sum of the independent losses from the two tasks. The constants’ x and y values range between 0 to 1 to set the loss weights depicting each task’s loss-share to the overall loss.
5 EXPERIMENTS, RESULTS, AND ANALYSIS
In this section, we discuss the experimental setup, report the evalu- ation results along with the necessary analysis.
5.1 Baselines
For a comprehensive evaluation of our proposed model, we com- pare its performance with the following baselines. All the models are developed for both single-task and multi-task variants under federated machine learning settings. • Bidirectional Long Short-term Memory+Att (BiLSTM+Att) [4]: This is a variant of simple LSTM model [15] which is capable of modelling contextual information from both for- ward and backward timesteps. We employ 128 neurons in the BiLSTM layer. The output from this layer is passed through an attention layer [3] which is followed by dense layer(s) and output layer(s) (one dense layer followed by an output layer for single-task and two task-specific dense layers followed by respective output layers for multi-task). Each dense layer employs 100 neurons. • Convolutional Neural Networks (CNN) [19]: This base- line constitutes of a single-layer CNN employing filter sizes of 3, 4, and 5 with 100 feature maps in each. MaxPooling op- eration is executed on the feature maps which are passed to the output layer with softmax activation for the predictions. The architecture is similar to the baseline model in [12]. The max pooled features are passed through dense layer(s) and output layer(s) as in BiLSTM+Att. • Bidirectional GRU+Att (BiGRU+Att) [2]: This model lever- ages a shared BiGRU [8] layer (128 units) and an attention layer to generate sentence representation which is passed through dense layer(s) and output layer(s) as in the BiL- STM+Att baseline. • CNN+LSTM [43]: In this stacked model, the CNN model extracts features from the text and the LSTM model inter- prets the features across timesteps. This model can extract both the global and local features of the text as well as the temporal interpretation. The output from the LSTM is passed through dense layer(s) and output layer(s) as in BiLSTM+Att. • Federated BERT-based Single Task Learning (Fed-BSTL): Besides the multi-task model, we build a BERT-based single- task model variant (Fed-BSTL) for the complaint task. The architecture is the same as the Fed-BMTL technique, except that there are no task-specific dense layers and output layers.
Instead, the sentence representation from the BERT model is passed to the output layer.
5.2 Experimental Setup
We implement our proposed framework and all the baselines on Python-based libraries, Keras11 [9] and Scikit-learn12 [27]. We per- form all the experiments on a single GeForce GTX 1080 Ti GPU and the average time taken to train the network was approximately 60 minutes. We report the mean accuracy and mean macro-F1 scores on the domain-specific federated test sets. For all the multi-task experiments (except GRU and LSTM), we weight the losses from the complaint and the sentiment task by 1 and 0.01, respectively. To find the optimal loss of weight constants we use the grid search method. For the multi-task GRU and LSTM models, we set the loss weights as 1 and 0.02 for the aforesaid tasks. Due to weakly supervised annotations of the sentiment labels, inherent losses in annotations could be more in this task than manually annotated sentiment labels. By setting lower loss weights for the sentiment task in comparison to the complaint detection task prevents the model from training fully on the loss from the sentiment task; thus avoiding the increased risk of improper weight modifications during the training. For both the datasets, We use the python-based keras-bert library for the BERT implementation. We fine-tune the ’bert-base’ model with the following set of hyperparameters: 𝛽1 = 0.9, 𝛽2 = 0.999, 𝜖=1e-7. We apply Grid Search13 technique to set the epoch size = 4 and learning rate = 3e-5 and train our network using Rectified- Adam [22]for the Complaints data. Similarly, for the Product Review dataset we set the epoch size = 4 and learning rate = 3e-5. We employ 𝑅𝑒𝐿𝑈activation in the dense layers and 𝑆𝑜𝑓𝑡𝑚𝑎𝑥activation in the output layers. In the GRU units we use 𝑇𝑎𝑛ℎactivation and and also apply a 𝑑𝑟𝑜𝑝𝑜𝑢𝑡[37] of 25% each. We employ a 𝑑𝑟𝑜𝑝𝑜𝑢𝑡of 25% following the attention layer as well as after every linear layer, to decrease the risk of overfitting. We clip and pad input sentences to set the maximum sequence length as 128. For the deep learning baselines we use pre-trained GloVe14 [28] word embedding which is trained on Common Crawl (840 billion tokens) corpus to get the word embedding representations. For the FL experiments, we varied the number of clients who participate in a single training round, we report the results with the worker size as 0.7 of the total number of workers for all the experiments.
5.3 Results and Discussion
We present the results for all our experiments in this section. Table 3 and 4 reports the results from the different experiments. For the Complaints data dataset, our proposed framework (Fed-BMTL) out- performs the single-task model (Fed-BSTL) for the complaint task with an improvement of 3.1 and 4.03 points in terms of accuracy and macro-F1 metrics, respectively. Moreover, it can be seen that the proposed BERT-based multi-task system exhibits improvements of 9.03 and 5.65 points in terms of macro-F1, over the best-performing deep learning multi-task baseline (BiGRU+Att) for both complaint
11https://keras.io/ 12https://scikit-learn.org/stable/ 13We experimented with epochs = [3,4,5] and learning rates = [1e-3, 2e-3, 3e-5] 14GloVe: http://nlp.stanford.edu/data/wordvecs/glove.840B.300d.zip
206
<!-- Source PDF page 7; coordinate-aware two-column reading order. -->
Full Paper HT ’21, August 30–September 2, 2021, Virtual Event, Ireland
Table 3: Overall classification results for the Complaints data dataset including the single-task variants, the proposed model and the baselines. Acc: Accuracy, F1: macro-F1 score for complaint prediction (± standard deviation). Bold face indicates leading results for each metric. The marks indicate these results are statistically significant. The results have been reported with communication rounds = 20.
Model STL MTL Complaint Complaint Sentiment Acc(%) F1(%) Acc(%) F1(%) Acc(%) F1(%)
SOTA[29] 80.5 78 - - - -
Deep learning Baselines BiLSTM+Att 79.96 ± .03 77.36 ± .03 81.63 ± .05 79.14 ± .04 63.54 61.43 CNN 73.21 ± .05 71.45 ± .03 77.89 ± .04 75 ± .04 62.33 59.6 BiGRU+Att 80.80 ± .03 77.64 ± .04 82.05 ± .04 79.78 ± .04 66.67 64 CNN+LSTM 77.43 ± .03 75.48 ± .04 77.26 ± .04 74.5 ± .05 60.5 57.08
BERT Baseline Fed-BSTL 85.44 ± .03 83 ± .05 - - - -
Proposed Approach Fed-BMTL - - 88.54 ± .04 87.03 ± .04 70.23 ± 0.5 69.65 ± 0.4
and sentiment tasks, respectively. In the case of Product Review dataset, the Fed-BMTL model outperforms the Fed-BSTL baseline by achieving the macro-F1 score of 70.27% for the primary task. Additionally, the proposed model also beats the classification re- sults of the best performing deep learning baseline BiGRU+Att model with an improvement of 1.24 and 0.42 points in terms of accuracy and macro-F1 metrics, respectively. The results signify that incorporation of a pre-trained model and knowledge about sen- timent in combination enhances the performance of the complaint identification task even under federated settings.
Centralized vs Federated framework: We also performed experi- ments under centralized settings for both the datasets. In the case of Complaints data, the Centralized BERT-based multi-task model (Cen-BMTL) achieved an accuracy of 89% and the macro-F1 score of 88.58% for the complaint task. For the sentiment task, the accuracy achieved is 71.57% and macro-F1 score of 70%. Whereas, in the case of Product Review dataset, the Cen-BMTL model achieved an accu- racy of 78.48% and the macro-F1 score is 77.02% for the primary task of complaint identification. It should be noted that the distribution of data across more than one client affects the performance of the federated learning framework which is not the case for models built on the server under centralized settings. We evaluate a schedule to test the Fed-BMTL model by varying the number of communication rounds, to find the optimal number of communication rounds when the model starts to converge. Figure 3 and 4 illustrate the relation between macro-F1 score and the num- ber of communication rounds for Fed-BSTL and Fed-BMTL frame- works, for the Complaints data and the Product Review dataset, respectively. It can be seen by round = 20 the Fed-BMTL model has converged.
Statistical Significance Test: The proposed approach of federated BERT-based framework is found to be statistically significant15
15We perform Student’s t-test for assessing the statistical significance
Figure 3: The plot of macro-F1 score against communication rounds for Fed-BSTL and Fed-BMTL frameworks in the com- plaint prediction task, for the Complaints data dataset. The Centralized learning macro-F1 score is 88.58%
above the following best performing model, Fed-BMTL, when tested against the null hypothesis with p-esteem 0.04.
5.4 Comparison with State-of-the-art System (SOTA)
We compare our proposed model with the state-of-the-art tech- niques [29] and [35] as reported in Table 3 and 4 . In the case of Complaints data, SOTA illustrates a thorough analysis of identi- fying complaints in social media by taking advantage of feature engineering techniques in a centralized setting. Their best predic- tive performance of 80.5% accuracy and 78% macro-F1 are attained using a logistic regression classifier with a variety of generic linguis- tic features and complaint-specific features. Whereas, in the case
207
<!-- Source PDF page 8; coordinate-aware two-column reading order. -->
Full Paper HT ’21, August 30–September 2, 2021, Virtual Event, Ireland
Table 4: The classification results for the Product Review dataset including single-task variants, the proposed model, and the baselines. Acc: Accuracy, F1: macro-F1 score for complaint prediction (± standard deviation). Boldface indicates leading results for each metric. The marks indicate these results are statistically significant. The results have been reported with communi- cation rounds = 20.
Model STL MTL Complaint Complaint Sentiment Acc(%) F1(%) Acc(%) F1(%) Acc(%) F1(%)
SOTA[35] - 68.38 - - - -
Deep learning Baselines BiLSTM+Att 66.18 ± .04 67.46 ± .06 69.58 ± .12 68.66 ± .04 49.54 48.43 CNN 59.11 ± .08 58.45 ± .13 61.89 ± .06 60 ± .04 46 45.33 BiGRU+Att 68 ± .05 66.78 ± .04 70 ± .08 69.85 ± .12 53.43 51.04 CNN+LSTM 62.77 ± .08 60.09 ± .12 65.59 ± .18 64.33 ± .12 49.22 48.38
BERT Baseline Fed-BSTL 69.56 ± 0.08 67.89 ± 0.14 - - - -
Proposed Approach Fed-BMTL - - 71.24 ± 0.5 70.27 ± 0.6 56.11 ± 0.6 55.65 ± 0.4
of the Product Review dataset, SOTA analyzes the dataset by per- forming multiple experiments using classical supervised machine learning classifiers, a deep learning classifier, and a semi-supervised graph-based classifier. Their deep learning model achieves the best F1-score of 68.38%. The reported results in Table 3 and 4 depict that our proposed system (Fed-BMTL) surpasses the state-of-the-art baseline for the complaint identification task. It should be noted that both the respective SOTA techniques were developed under centralized settings.
Figure 4: The plot of macro-F1 score against communication rounds for Fed-BSTL and Fed-BMTL frameworks in the com- plaint prediction task, for the Product Review dataset. The Centralized learning macro-F1 score is 77.02%
5.5 Error Analysis
In this section, we report the most commonly occurring errors that our proposed model faces.
• Imbalanced Dataset: The imbalanced distribution of the com- plaint dataset affects the predictions of the proposed model. For example, "I applied for a job at a local store and nobody has gotten back to me on anything. Can you guys look into it?". Predicted class: non-complaint. For the given example the correct class is complaint but due to the skewed data distri- bution, the model incorrectly predicts it as non-complaint. • Sentiment Incongruity: Instances consisting of ironical com- ments where the underlying tone may be negative but the sentence is of non-complaint type, the proposed Fed-BMTL framework incorrectly predicts such instances as complaint. For example, "Fuck me live chat with BT and Vodafone is worse than the VWR!". One of the reasons could be the usage of explicit negative words (such as rage, annoyance) and in- tensifiers (very, too, etc.) used while expressing anger or disapproval online. On the other hand, positive words are less explicit which leads to incorrect classification. Another reason could be the weak-supervision-based sentiment label- ing performed to get the sentiment labels for the Complaints dataset. • Common Phrases: Sentences which are commonly used for querying the customer support with varying sentiments makes the model susceptible to errors. For example "How can I get a hold of you so we can discuss the problem I am having with my coat?" and "I need to get in contact with someone regarding the fc.com website. I work for a major search engine. Tried enquiries@ and webform". For the first sentence, the predicted class is complaint. But the correct class is non- complaint, for the given example. Whereas for the second sentence the model correctly predicts as complaint class.
208
<!-- Source PDF page 9; coordinate-aware two-column reading order. -->
Full Paper HT ’21, August 30–September 2, 2021, Virtual Event, Ireland
Due to lack of data and diversity in the data, the model has picked up the most recurrent class. • Data Partitioning: In federated learning configuration, the client’s workload decides the performance of the federated learning system. Even in this case, there are some domains with a very small amount of data samples that affects the performance of the proposed framework. • Translation Errors: In the case of the Hindi Product Review dataset we used Google translate to get the English translated sentences specifically for sentiment tagging. Due to this few of the translation of the instances is incorrect affecting the performance of the proposed model.
6 CONCLUSION AND FUTURE WORK
In this paper, we illustrate the feasibility of a federated learning framework in the area of complaint identification based on different domains where privacy, locality of data, and ownership guarantees are preserved. It should be noted that complaint tweets constituting different domains in our work resemble the grievances accumu- lated by different organizations in real life. We have proposed a BERT-based multi-task system for complaint and sentiment classi- fication tasks in a federated learning scenario. On comprehensive evaluation based on two real-world datasets, we observe that our proposed approach has outperformed the baselines significantly. For the English Complaints data the proposed model obtained an 87.03% macro-F1 score for complaint task (primary task) and 69.65% macro-F1 score for sentiment analysis (auxiliary task). Furthermore, for the Product Review dataset the proposed model achieves an ac- curacy of 71.24% and macro-F1 score of 70.27% for the primary task of complaint identification, surpassing the existing state-of-the-art results. While federated learning presents practical privacy advantages, im- plementing stronger confidentiality guarantees through differential privacy [1] in the given scenario is an exciting direction for future work. We also intend to explore other modalities such as images in complaint identification. We also intend to explore other relevant contributing factors to complaint identification from social media data such as politeness marker in texts and other modalities such as images. Acknowledgement: This publication is an outcome of the R&D work undertaken in the project under the Visvesvaraya Ph.D. Scheme of Ministry of Electronics & Information Technology, Government of India, being implemented by Digital India Corporation (Formerly Media Lab Asia).
REFERENCES
[1] Martín Abadi, Andy Chu, Ian J. Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. Deep Learning with Differential Privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Commu- nications Security, Vienna, Austria, October 24-28, 2016, Edgar R. Weippl, Stefan Katzenbeisser, Christopher Kruegel, Andrew C. Myers, and Shai Halevi (Eds.). ACM, 308–318. https://doi.org/10.1145/2976749.2978318 [2] Shad Akhtar, Deepanway Ghosal, Asif Ekbal, Pushpak Bhattacharyya, and Sadao Kurohashi. 2019. All-in-one: Emotion, sentiment and intensity prediction using a multi-task ensemble framework. IEEE Transactions on Affective Computing (2019). [3] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.). http://arxiv.org/abs/1409.0473
[4] Xuemei Bai. 2018. Text classification based on LSTM and attention. In 2018 Thirteenth International Conference on Digital Information Management (ICDIM), Berlin, Germany, September 24-26, 2018. IEEE, 29–32. https://doi.org/10.1109/ ICDIM.2018.8847061 [5] Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H. Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. 2017. Practi- cal Secure Aggregation for Privacy-Preserving Machine Learning. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, CCS 2017, Dallas, TX, USA, October 30 - November 03, 2017, Bhavani M. Thurais- ingham, David Evans, Tal Malkin, and Dongyan Xu (Eds.). ACM, 1175–1191. https://doi.org/10.1145/3133956.3133982 [6] Rich Caruana. 1997. Multitask learning. Machine learning 28, 1 (1997), 41–75. [7] Rich Caruana and Virginia R. de Sa. 1996. Promoting Poor Features to Su- pervisors: Some Inputs Work Better as Outputs. In Advances in Neural In- formation Processing Systems 9, NIPS, Denver, CO, USA, December 2-5, 1996, Michael Mozer, Michael I. Jordan, and Thomas Petsche (Eds.). MIT Press, 389– 395. http://papers.nips.cc/paper/1231-promoting-poor-features-to-supervisors- some-inputs-work-better-as-outputs [8] KyungHyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the Properties of Neural Machine Translation: Encoder-Decoder Ap- proaches. CoRR abs/1409.1259 (2014). arXiv:1409.1259 http://arxiv.org/abs/1409. 1259 [9] François Chollet et al. 2015. keras. [10] Kristof Coussement and Dirk Van den Poel. 2008. Improving customer complaint management by automatic email classification using linguistic style features as predictors. Decision Support Systems 44, 4 (2008), 870–882. [11] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). Association for Computa- tional Linguistics, 4171–4186. https://doi.org/10.18653/v1/n19-1423 [12] Jiachen Du, Lin Gui, Ruifeng Xu, and Yulan He. 2017. A Convolutional Attention Model for Text Classification. In Natural Language Processing and Chinese Com- puting - 6th CCF International Conference, NLPCC 2017, Dalian, China, November 8-12, 2017, Proceedings (Lecture Notes in Computer Science, Vol. 10619), Xuanjing Huang, Jing Jiang, Dongyan Zhao, Yansong Feng, and Yu Hong (Eds.). Springer, 183–195. https://doi.org/10.1007/978-3-319-73618-116 [13] Avishek Ghosh, Justin Hong, Dong Yin, and Kannan Ramchandran. 2019. Robust Federated Learning in a Heterogeneous Environment. CoRR abs/1906.06629 (2019). arXiv:1906.06629 http://arxiv.org/abs/1906.06629 [14] Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. 2017. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour. CoRR abs/1706.02677 (2017). arXiv:1706.02677 http://arxiv.org/abs/1706.02677 [15] Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9, 8 (1997), 1735–1780. [16] Li Huang, Yifeng Yin, Zeng Fu, Shifa Zhang, Hao Deng, and Dianbo Liu. 2020. LoAdaBoost: Loss-based AdaBoost federated machine learning with reduced computational complexity on IID and non-IID intensive care data. Plos one 15, 4 (2020), e0230706. [17] Clayton J. Hutto and Eric Gilbert. 2014. VADER: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text. In Proceedings of the Eighth International Conference on Weblogs and Social Media, ICWSM 2014, Ann Arbor, Michigan, USA, June 1-4, 2014, Eytan Adar, Paul Resnick, Munmun De Choudhury, Bernie Hogan, and Alice H. Oh (Eds.). The AAAI Press. http://www.aaai.org/ ocs/index.php/ICWSM/ICWSM14/paper/view/8109 [18] Eunjeong Jeong, Seungeun Oh, Hyesung Kim, Jihong Park, Mehdi Bennis, and Seong-Lyun Kim. 2018. Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data. CoRR abs/1811.11479 (2018). arXiv:1811.11479 http://arxiv.org/abs/1811.11479 [19] Yoon Kim. 2014. Convolutional Neural Networks for Sentence Classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Doha, Qatar, 1746–1751. https://doi.org/10.3115/v1/D14-1181 [20] Abhishek Kumar, Asif Ekbal, Daisuke Kawahara, and Sadao Kurohashi. 2019. Emotion helps Sentiment: A Multi-task Model for Sentiment and Emotion Anal- ysis. In International Joint Conference on Neural Networks, IJCNN 2019 Budapest, Hungary, July 14-19, 2019. IEEE, 1–8. https://doi.org/10.1109/IJCNN.2019.8852352 [21] M Lailiyah, S Sumpeno, and IK E Purnama. 2017. Sentiment analysis of public complaints using lexical resources between Indonesian sentiment lexicon and Sentiwordnet. In 2017 International Seminar on Intelligent Technology and Its Applications (ISITIA). IEEE, 307–312. [22] Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2020. On the Variance of the Adaptive Learning Rate and Beyond. In 8th International Conference on Learning Representations, ICLR 2020,
209
<!-- Source PDF page 10; coordinate-aware two-column reading order. -->
Full Paper HT ’21, August 30–September 2, 2021, Virtual Event, Ireland
Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. https://openreview. net/forum?id=rkgz2aEKDr [23] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. 2017. Communication-Efficient Learning of Deep Net- works from Decentralized Data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS 2017, 20-22 April 2017, Fort Laud- erdale, FL, USA (Proceedings of Machine Learning Research, Vol. 54), Aarti Singh and Xiaojin (Jerry) Zhu (Eds.). PMLR, 1273–1282. http://proceedings.mlr.press/ v54/mcmahan17a.html [24] H. Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. 2018. Learning Differentially Private Recurrent Language Models. In 6th Interna- tional Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net. https: //openreview.net/forum?id=BJ0hF1Z0b [25] E Olshtain and L Weinbach. 1985. Complaints: A Study of Speech Act Behavior among Native and Nonnative Speakers of Hebrew. The Prag-matic Perspective. [26] Sachin Pawar, Nitin Ramrakhiyani, Girish K. Palshikar, and Swapnil Hingmire. 2015. Deciphering Review Comments: Identifying Suggestions, Appreciations and Complaints. In Natural Language Processing and Information Systems - 20th International Conference on Applications of Natural Language to Information Systems, NLDB 2015 Passau, Germany, June 17-19, 2015 Proceedings (Lecture Notes in Computer Science, Vol. 9103), Chris Biemann, Siegfried Handschuh, André Freitas, Farid Meziane, and Elisabeth Métais (Eds.). Springer, 204–211. https://doi.org/10.1007/978-3-319-19581-018 [27] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in Python. Journal of machine learning research 12, Oct (2011), 2825–2830. [28] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, Alessandro Moschitti, Bo Pang, and Walter Daelemans (Eds.). ACL, 1532–1543. https://doi.org/10.3115/v1/d14-1162 [29] Daniel Preotiuc-Pietro, Mihaela Gaman, and Nikolaos Aletras. 2019. Automati- cally Identifying Complaints in Social Media. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, Anna Korhonen, David R. Traum, and Lluís Màrquez (Eds.). Association for Computational Linguistics, 5008–5019. https://doi.org/10.18653/v1/p19-1495 [30] Syed Arbaaz Qureshi, Gael Dias, Mohammed Hasanuzzaman, and Sriparna Saha. 2020. Improving depression level estimation by concurrently learning emotion intensity. IEEE Computational Intelligence Magazine 15, 3 (2020), 47–59. [31] Syed Arbaaz Qureshi, Sriparna Saha, Mohammed Hasanuzzaman, and Gaël Dias. 2019. Multitask Representation Learning for Multimodal Estimation of Depres- sion Level. IEEE Intelligent Systems 34, 5 (2019), 45–52. [32] Apoorva Singh and Sriparna Saha. 2021. Are You Really Complaining? A Multi- task Framework for Complaint Identification, Emotion and Sentiment Classifi- cation. In Proceedings of the 16th International Conference on Document Analysis
and Recognition. Springer, Accepted. [33] Apoorva Singh, Sriparna Saha, Md Hasanuzzaman, and Kuntal Dey. 2021. Mul- titask learning for complaint identification and sentiment analysis. Cognitive Computation (2021), 1–16. [34] Apoorva Singh, Sriparna Saha, Md Hasanuzzaman, and Anubhav Jangra. 2021. Identifying Complaints based on Semi-Supervised Mincuts. In Expert System with Applications. Elsevier, Accepted. [35] Raghvendra Pratap Singh, Rejwanul Haque, Mohammed Hasanuzzaman, and Andy Way. 2020. Identifying Complaints from Product Reviews: A Case Study on Hindi. In Proceedings of The 28th Irish Conference on Artificial Intelligence and Cognitive Science, Dublin, Republic of Ireland, December 7-8, 2020 (CEUR Workshop Proceedings, Vol. 2771), Luca Longo, Lucas Rizzo, Elizabeth Hunter, and Arjun Pakrashi (Eds.). CEUR-WS.org, 217–228. http://ceur-ws.org/Vol- 2771/AICS2020paper28.pdf [36] Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet S. Talwalkar. 2017. Federated Multi-Task Learning. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (Eds.). 4424–4434. https://proceedings.neurips.cc/paper/2017/hash/ 6211080fa89981f66b1a0c9d55c61d0f-Abstract.html [37] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research 15, 1 (2014), 1929–1958. [38] Suhatati Tjandra, Amelia Alexandra Putri Warsito, and Judi Prajetno Sugiono. 2015. Determining citizen complaints to the appropriate government departments using KNN algorithm. In 2015 13th International Conference on ICT and Knowledge Engineering (ICT & Knowledge Engineering 2015). IEEE, 1–4. [39] Camilla Vásquez. 2011. Complaints online: The case of TripAdvisor. Journal of Pragmatics 43, 6 (2011), 1707–1717. [40] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. CoRR abs/1706.03762 (2017). arXiv:1706.03762 http://arxiv.org/abs/ 1706.03762 [41] Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. 2019. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST) 10, 2 (2019), 1–19. [42] Wei Yang, Luchen Tan, Chunwei Lu, Anqi Cui, Han Li, Xi Chen, Kun Xiong, Muzi Wang, Ming Li, Jian Pei, and Jimmy Lin. 2019. Detecting Customer Com- plaint Escalation with Recurrent Neural Networks and Manually-Engineered Features. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 2 (Industry Pa- pers), Anastassia Loukina, Michelle Morales, and Rohit Kumar (Eds.). Association for Computational Linguistics, 56–63. https://doi.org/10.18653/v1/n19-2008 [43] Chunting Zhou, Chonglin Sun, Zhiyuan Liu, and Francis C. M. Lau. 2015. A C-LSTM Neural Network for Text Classification. CoRR abs/1511.08630 (2015). arXiv:1511.08630 http://arxiv.org/abs/1511.08630
210
Visual and table regions
The following coordinate-derived renders preserve the complete visual/table regions from the source pages; their captions are retained in the full text above.
Table 1. Table 1: Data distribution in the Complaints-Data.
Table 2. Table 2: Data distribution in the Product Review dataset
Figure 1. Figure 1: Flowchart of the Federated Learning framework.
Figure 2. Figure 2: BERT-based Multi-task Model Architecture
Table 3. Table 3: Overall classification results for the Complaints data dataset including the single-task variants, the proposed model and the baselines. Acc: Accuracy, F1: macro-F1 score for complaint prediction (± standard deviation). Bold face indicates leading results for each metric. The marks indicate these results are statistically significant. The results have been reported with communication rounds = 20.
Figure 3. Figure 3: The plot of macro-F1 score against communication rounds for Fed-BSTL and Fed-BMTL frameworks in the com- plaint prediction task, for the Complaints data dataset. The Centralized learning macro-F1 score is 88.58%
Table 4. Table 4: The classification results for the Product Review dataset including single-task variants, the proposed model, and the baselines. Acc: Accuracy, F1: macro-F1 score for complaint prediction (± standard deviation). Boldface indicates leading results for each metric. The marks indicate these results are statistically significant. The results have been reported with communi- cation rounds = 20.
Figure 4. Figure 4: The plot of macro-F1 score against communication rounds for Fed-BSTL and Fed-BMTL frameworks in the com- plaint prediction task, for the Product Review dataset. The Centralized learning macro-F1 score is 77.02%
Do you like what you are reading? Subscribe to receive updates.
Unsubscribe anytime