An IDE Plugin for Gamified Continuous Integration
Philipp Straubinger
University of Passau
Passau, Germany
Gordon Fraser
University of Passau
Passau, Germany
ABSTRACT
Interruptions and context switches resulting from meetings, urgent
tasks, emails, and queries from colleagues contribute to productivity
losses in developers’ daily routines. This is particularly challeng-
ing for tasks like software testing, which are already perceived
as less enjoyable, prompting developers to seek distractions. To
mitigate this, applying gamication to testing activities can en-
hance motivation for test writing. One such gamication tool is
Gamekins
, which integrates challenges, quests, achievements, and
leaderboards into the Jenkins CI (continuous integration) platform.
However, as
Gamekins
is typically accessed through a browser, it
introduces a context switch. This paper presents an IntelliJ plugin
designed to seamlessly integrate
Gamekins
’ gamication elements
into the IDE, aiming to minimize context switches and boost devel-
oper motivation for test writing.
CCS CONCEPTS
• Software and its engineering
→
Software testing and debug-
ging; Integrated and visual development environments.
KEYWORDS
Gamication, IDE, IntelliJ, Software Testing, Continuous Integra-
tion
ACM Reference Format:
Philipp Straubinger and Gordon Fraser. 2024. An IDE Plugin for Gamied
Continuous Integration. In 2024 First IDE Workshop (IDE ’24), April 20, 2024,
Lisbon, Portugal. ACM, New York, NY, USA, 4 pages. https://doi.org/10.1145/
3643796.3648462
1 INTRODUCTION
Gamekins
[
3
] is a tool that integrates gamication into the soft-
ware testing process within the Jenkins continuous integration
platform, to encourage developers to improve their testing prac-
tices. Addressing the challenge of achieving high software quality
and recognizing the common lack of motivation for testing among
developers,
Gamekins
incorporates gamication elements like chal-
lenges, quests, leaderboards, and achievements. Developers earn
points by completing test-related challenges and quests, engag-
ing in leaderboard competition, and seeking achievements, with
the integration into Jenkins aiming to make gamication easily
accessible to developers without additional training or resources.
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for prot or commercial advantage and that copies bear this notice and the full citation
on the rst page. Copyrights for components of this work owned by others than the
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specic permission
and/or a fee. Request permissions from [email protected].
IDE ’24, April 20, 2024, Lisbon, Portugal
© 2024 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 979-8-4007-0580-9/24/04.. . $15.00
https://doi.org/10.1145/3643796.3648462
In a recent study [
4
] we integrated
Gamekins
into an undergrad-
uate software testing course to motivate students to engage more
with testing. This study revealed a positive correlation between
students’ testing behavior and the use of
Gamekins
, resulting in
a signicant improvement in correct results compared to a non-
gamied cohort. Additionally, students expressed satisfaction with
the tool, conrming its eectiveness as a teaching tool.
However, a notable issue reported by the students in this ex-
periment was the necessity to switch from their Integrated De-
velopment Environment (IDE) to a browser with
Gamekins
open,
causing a productivity dip and time loss navigating to Jenkins.
Such context switches, which represent a known productivity chal-
lenge [
2
], are a concern not only for students but also for developers,
who spend approximately 50% of their work time in the IDE [1].
In other work, we established that gamication of testing is pos-
sible directly in the IDE. Our
IntelliGame
plugin for the
IntelliJ
Java IDE
1
implements a multi-level achievement system rewarding
positive testing behavior [
5
]. In a controlled experiment with 49 par-
ticipants
IntelliGame
demonstrated clear improvements in testing
behavior, for example, causing users to write more tests, achieve
higher coverage and mutation scores, run tests more frequently,
and achieve functionality earlier.
In this paper, we aim to use the experience of gamifying testing
directly in the IDE to address the problem of context switches
caused by gamication in the CI. We therefore developed a new
plugin for the
IntelliJ
Java IDE that seamlessly integrates the
functionality of
Gamekins
directly into the IDE, oering enhanced
features to improve developers’ workow.
2 GAMIFICATION ELEMENTS OF GAMEKINS
2.1 Challenges
Gamekins
oers a variety of dierent challenges, which are test
and quality-oriented tasks for the developer to solve. There are
currently eight dierent types of challenges available
2
:
•
Build Challenge: This challenge shows the developer that
the build on the CI failed and has to be xed.
•
Test Challenge: This generic challenge shows that the de-
veloper has to write a new test.
•
Class Coverage Challenge: This challenge tasks the devel-
oper to cover more lines in a specic class.
•
Method Coverage Challenge: This challenge focuses on
improving the coverage of a specic method.
•
Line Coverage Challenge: This challenge assigns the de-
veloper an uncovered line that they have to cover.
•
Branch Coverage Challenge: This challenge focuses on
the improvement of branch coverage of a covered line.
1
https://www.jetbrains.com/idea/
2
Detailed information can be found in prior work [3, 4]
arXiv:2403.03557v1 [cs.SE] 6 Mar 2024
IDE ’24, April 20, 2024, Lisbon, Portugal Straubinger et al.
•
Mutation Challenge: This challenge tasks the developer
to detect a mutant generated by PIT
3
by writing a new test.
•
Smell Challenge: This challenge focuses on the removal of
code and test smell with the help of SonarLint
4
.
2.2 Quests
Since the original inception as described in prior work [
3
,
4
], the way
quests are implemented has changed. Instead of having multiple
challenges solved after each other, quests are now test- and quality-
related tasks that involve making multiple quality improvements
and interacting with gamication elements. It is therefore not bound
to challenges anymore, but rather on improving dierent test and
quality-related metrics. The following types of quests are available:
•
Add Tests Quest: This quest tasks the developer to add a
specied number of tests to the existing test suite.
•
Cover Branches Quest: This quest focuses on covering an
additional specied number of branches with new tests.
•
Cover Lines Quest: This quest focuses on covering an ad-
ditional specied number of lines with new tests.
•
Receive Challenges Quest: This quest promotes interac-
tion with other developers by receiving a challenge from
another developer.
•
Send Challenges Quest: This quest promotes interaction
with other developers by sending a challenge to them.
•
Solve Achievements Quest: This quest focuses on the com-
pletion of a new achievement.
•
Solve Challenges Quest: This quest tasks the developer to
solve a specied number of new challenges of one type.
•
Solve Challenges Without Rejection Quest: This quest
focuses on solving a specied number of challenges regard-
less of their types without rejecting one in between.
2.3 Achievements
Developers are rewarded based on their testing accomplishments,
encompassing various achievements of diering diculty levels.
These achievements range from simple tasks like adding a test to
a project to more challenging objectives such as achieving 100%
coverage. Certain achievements remain hidden until they are com-
pleted and are tied to individual actions, like the addition of new
tests. Achievements are triggered when users commit to the project
repository. There are two main types of achievements: individ-
ual achievements, which involve completing a specic number of
challenges, and project-level achievements, which are linked to
reaching a particular threshold within the project.
3 INTELLIJ PLUGIN FOR GAMEKINS
The Gamekins IntelliJ plugin can be installed on any
IntelliJ
Community Edition or Ultimate IDE, adding a new tab on the right
side of the IDE interface. Upon initial access to this tab, a login
prompt will appear, prompting users to enter their username, pass-
word, URL, and project name to establish a connection with the rel-
evant Jenkins instance. Subsequent logins are streamlined through
the use of a token, eliminating the need for repeated manual logins,
although users have the exibility to log out as needed. Project
3
https://pitest.org/
4
https://www.sonarsource.com/products/sonarlint/
and user congurations are centrally managed within
Gamekins
on a Jenkins server rather than within the Gamekins IntelliJ
plugin. This allows for extensive collaboration within the project,
and the analyses conducted by
Gamekins
need not be executed
locally. A customized API integrated into
Gamekins
empowers the
plugin to fetch challenges, quests, and user data from
Gamekins
,
as well as transmit data to it. The Gamekins IntelliJ plugin tab
encompasses various pages presenting all the information and gami-
cation elements generated and provided by
Gamekins
, along with
the buttons on the bottom to change between these pages (Fig. 1).
3.1 Challenges
All challenges are visible in the Gamekins IntelliJ plugin, with
the availability of dierent information and actions based on the
challenge type. For instance, a Test Challenge solely displays in-
formation about the challenge without further interactions with
IntelliJ
(Fig. 1). Similarly, a Build Challenge provides information
and includes an additional link to the log les in Jenkins. The
Class and Method Coverage Challenges oer a button to navigate
to and highlight the respective class or method. This feature is also
available for Line and Branch Coverage challenges, highlighting
the specied line. The Branch Coverage Challenge additionally pro-
vides information about covered and uncovered branches for the
tested line. The Mutation and Smell Challenges allow developers
to go to and highlight the original line of code or the start of the
test smell, respectively. These challenges also have an expandable
area containing details such as the mutant or a smell description.
The challenges are not only visible in the Gamekins IntelliJ
plugin tab but also feature yellow highlighting in the source les,
aiding developers in easily identifying challenges in their current
code. Hovering over the highlighted code snippet reveals a tooltip
with the challenge description and a button to open the challenges
page in the Gamekins IntelliJ plugin (Fig. 2). This page displays
current challenges along with completed and rejected ones (Fig. 1).
Developers can reject challenges with an explanation if they
deem them irrelevant, prompting the generation of new challenges.
Once rejected, challenges cannot be reinstated, except for Class
Coverage Challenges, which can be restored using the Undo button
(Fig. 1). If a developer nds a challenge interesting but unsuitable
for the current workow, they can store a congurable number
of challenges. These stored challenges can also be shared with a
colleague working on the same project, especially if the source code
is segmented into specialized elds unknown to the participant.
To address a challenge, developers simply need to commit and
push their changes to their version control system, triggering Jenk-
ins to run the CI pipeline in the background. Developers can seam-
lessly continue their work in the project as
Gamekins
automatically
executes after the pipeline concludes, providing notications upon
completion. General notications about the nished build and spe-
cic notications for each solved and generated challenge or quest
are displayed (Fig. 3) in
IntelliJ
. These notications include links to
view the corresponding elements in the Gamekins IntelliJ plugin.
An IDE Plugin for Gamified Continuous Integration IDE ’24, April 20, 2024, Lisbon, Portugal
Figure 1: Current, completed and rejected challenges
Figure 2: Highlighted line of code with the tooltip giving
information about the challenge
Figure 3: Notication received after solving a challenge
3.2 Quests
Each quest in the Gamekins IntelliJ plugin is displayed on the
quests page (Fig. 4) along with its description and a progress bar.
Both ongoing and completed quests are shown. The progress bar
indicates the percentage of quest completion and is dynamically
updated whenever relevant actions are undertaken.
IDE ’24, April 20, 2024, Lisbon, Portugal Straubinger et al.
Figure 4: Current and completed quests
Figure 5: List of achievements
3.3 Achievements
Presently, the Gamekins IntelliJ plugin encompasses all achieve-
ments from
Gamekins
, allowing developers to access a dedicated
tab to view both acquired and available achievements (Fig. 5). Each
achievement consists of an icon, a title, a description, and the date
and time it was awarded.
3.4 Leaderboards
To incentivize developers to solve challenges and complete quests,
points are awarded based on the task’s diculty. These points are
accumulated for each user and team, prominently displayed in a
dedicated leaderboard within the Gamekins IntelliJ plugin tab
Figure 6: Example leaderboard
(Fig. 6). The leaderboard showcases rankings for individual users
as well as teams, emphasizing their scores in the logged-in project.
Information regarding completed challenges, quests, achievements,
and earned points is presented on each leaderboard. Additionally,
users have the option to personalize their appearance by selecting
one of 50 avatars available within Gamekins.
4 CONCLUSIONS
The Gamekins IntelliJ plugin seamlessly incorporates the func-
tionalities of
Gamekins
into the IDE, presenting essential informa-
tion such as leaderboards, challenges, quests, and achievements di-
rectly within the developer’s view. This integration minimizes con-
text switches and associated productivity losses. We are currently
enhancing the achievements system of
Gamekins
and introducing
new ones. These improvements, along with future enhancements,
will be integrated into Gamekins IntelliJ plugin. Additionally, we
plan to assess the plugin in our university courses to evaluate its ef-
fectiveness. Lastly, we aim to introduce new types of challenges and
achievements, building upon our experience with IntelliGame.
Gamekins IntelliJ plugin is available at:
https://github.com/se2p/Gamekins-IntelliJ-Plugin
5 ACKNOWLEDGEMENTS
This work is supported by the DFG under grant FR 2955/2-1
REFERENCES
[1]
André N. Meyer, Earl T. Barr, Christian Bird, and Thomas Zimmermann. 2021.
Today Was a Good Day: The Daily Life of Software Developers. IEEE Trans.
Software Eng. 47, 5 (2021), 863–880. https://doi.org/10.1109/TSE.2019.2904957
[2]
André N. Meyer, Thomas Fritz, Gail C. Murphy, and Thomas Zimmermann. 2014.
Software developers’ perceptions of productivity. In Proceedings of the 22nd ACM
SIGSOFT International Symposium on Foundations of Software Engineering, (FSE-22),
Hong Kong, China, November 16 - 22, 2014, Shing-Chi Cheung, Alessandro Orso,
and Margaret-Anne D. Storey (Eds.). ACM, 19–29. https://doi.org/10.1145/2635868.
2635892
[3]
Philipp Straubinger and Gordon Fraser. 2022. Gamekins: Gamifying Software Test-
ing in Jenkins. In 44th IEEE/ACM International Conference on Software Engineering:
Companion Proceedings, ICSE Companion 2022, Pittsburgh, PA, USA, May 22-24,
2022. ACM/IEEE, 85–89. https://doi.org/10.1145/3510454.3516862
[4]
Philipp Straubinger and Gordon Fraser. 2024. Gamifying a Software Testing
Course with Continuous Integration. In 46th IEEE/ACM International Conference
on Software Engineering, ICSE 2024, Lisbon, Portugal, May 14-20, 2024. https:
//doi.org/10.1145/3639474.3640054
[5]
Philipp Straubinger and Gordon Fraser. 2024. Improving Testing Behavior by
Gamifying IntelliJ. In Proceedings of the 46th IEEE/ACM International Conference
on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024. ACM, 49:1–
49:13. https://doi.org/10.1145/3597503.3623339
