Call For Paper October - 2023
Submit Online
INTERNATIONAL JOURNAL OF INFORMATION TECHNOLOGY (IJIT) > Volume 4, Issue 2, July-December 2023 PDF HTML XML
SOLVING COMPLEXITY AND IMPROVING STORAGE EFFICIENCY IN MANAGING ELASTICSEARCH CLUSTERS USING KUBERNETES AND ECK OPERATORS
Amreth Chandrasehar 1Article Id - IJIT_4_02_001, Pages : 1-9, Date of Publication : 01, August 2023
Google Scholar Link
Academia Link
Scopedatabase Link : https://sdbindex.com/Document/document_search?title=SOLVING COMPLEXITY AND IMPROVING STORAGE EFFICIENCY IN MANAGING ELASTICSEARCH CLUSTERS USING KUBERNETES AND ECK OPERATORS&type=1
Abstract
Elasticsearch is a popular open-source tool based on Apache Lucene, widely used for search, storing large amounts of data. It is being used in many small startups to large enterprises for Full-text search, log analytics, application search, geospatial data analysis and very recently also used in SIEM. As business needs grows, the Elasticsearch cluster needs to be scaled and often, the operators of Elasticsearch face many complex issues to manage the cluster. In this article, solution is provided on how to reliably manage the clusters at petabyte scale. Also, key information on monitoring, alerting and autoscaling the clusters are provided for the operators of the clusters to easily manage large scale Elasticsearch clusters. This paper aims to provide key information to the administrators and users of Elasticsearch clusters to reliably use to handle large volumes of data and possibly used as a Data Lake. It will also help organizations to have better financial controls, operational efficiency and do more with less by adopting key solutions discussed in this paper.
Keywords
Elasticsearch, ELK, Elastic Stack, Observability, SIEM, Machine Learning, Kubernetes, ECK Operator, Data Lake, Cost Control
Share and Cite:
References
[1] Elasticsearch on Kubernetes: Step-by-step guides to run ELK on the most popular k8s platforms. Available: https://portworx.com/elasticsearch-kubernetes/
[2] Deepak Singh Dhami, When to use Kubernetes operators vs. Helm charts, Published: 01 Sep 2021. Available: https://www.techtarget.com/searchitoperations/tip/When-to-use-Kubernetes-operators-vs-Helm-charts?Offer=abt_pubpro_AI-Insider
[3] A simplified stack monitoring experience in Elastic Cloud on Kubernetes, 16 Sep 202. Available: https://www.elastic.co/blog/a-simplified-stack-monitoring-experience-in-elastic-cloud-on-kubernetes
[4] Migrate an ELK Stack to Elastic Cloud on AWS. Available: https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/migrate-an-elk-stack-to-elastic-cloud-on-aws.html
[5] Dirk Michel, On Amazon EKS and cluster add-ons, 5 Jul 2022. Available: https://medium.com/@micheldirk/on-running-amazon-eks-cluster-add-ons-29fae658e902
[6] Kevin Coleman, Apoorva Kulkarni, Mikhail Shapirov, and Vara Bonthu, Bootstrapping clusters with EKS Blueprints, 20 APR 2022. Available: https://aws.amazon.com/blogs/containers/bootstrapping-clusters-with-eks-blueprints/
[7] Adam Quan, Getting started with Elastic Cloud on Kubernetes: Deployment, 04 Sep 2019. Available: https://www.elastic.co/blog/getting-started-with-elastic-cloud-on-kubernetes-deployment
[8] Brandon Mensing, Peter Kim, Part 2.0: The true story behind Elasticsearch storage requirements, 15 Sep 2015. Available: https://www.elastic.co/blog/elasticsearch-storage-the-true-story-2.0
[9] Christian Dahlqvist, Filebeat modules, access logs and Elasticsearch storage requirements, 19 Jun 2019. Available: https://www.elastic.co/blog/filebeat-modiles-access-logs-and-elasticsearch-storage-requirements
Indexing
Scope Database
Our IAEME Publication journals are indexed in various indexing search engines
- Scope Database (www.sdbindex.com)
- Google Scholar (https://scholar.google.com)
- ResearchGate (https://www.researchgate.net/)
- Academia (https://www.academia.edu/)
- SSRN (https://www.ssrn.com/)
- Indian Citation Index (ICI) (http://www.indiancitationindex.com/)
- Advanced Science Index (https://journal-index.org/)
- ResearchBib (https://www.researchbib.com/)
Aim and Scope
International Journal of Information Technology (IJIT) is a top-ranked journal, focused on new research addressing information, management, and communications technologies as applied to the digital worlds of business, government and non-governmental enterprises. The Journal is especially interested in studies that address emerging technologies and their convergence including SMAC (social media, mobile, analytics, cloud) and BRAID (blockchain, robotics, automation of knowledge work, internet of things and digital fabrication).
Chief Editor
Dr. A.Udayakumar - Hindustan University, India
Managing Editor
Dr. Dheyaa J. Kadhim - College of Engineering, University of Baghdad, Iraq
Associate Editor
Dr. V.Antony Joe Raja - Sri Muthukumaran Institute of Technology, India
Prof. B. Arthi Gandhimathi - IAEME Publication, India
Dr. N. Tamil Selvan - IAEME Publication, India
Editorial Board
Dr. Preeti Patil - Head and Associate Professor, Department of Information Technology, D.Y. Patil College of Engineering, Akurdi, Pune, India
Dr. Preeti Patil - Head and Associate Professor, Department of Information Technology, D.Y. Patil College of Engineering, Akurdi, Pune, India
Dr. SURYAKANT B PATIL - JSPM's ICOER, Wagholi, Pune –MS
Dr. Satinderjeet Singh - Associate Manager, The Children’s Place, 500 Plaza Drive, Secaucus, New Jersey, USA – 07094
Mr. Rudrendu Paul - Boston University, USA
Reviewer Board
Prof. Muhammad Naufal Bin Mansor - University Malaysia Perlis, Malaysia
Dr. Harman Preet Singh - PGDAV College, India
Dr. M. Giri - Sreenivasa Institute of Technology and Management Studies, India
Dr. R.Senthil Kumar - Sri Nandhanam College Of Engineering and Technology,Tirupattur,Vellore
Dr. Chitra Kiran .N - Sai Vidya Institute of Technology, Bangalore
Dr. Shrikant Tiwari - Shri Shankaracharya Group of Institutions- Chattisgarh
Dr. H.S.Behera - Veer Surendra Sai University of Technology (VSSUT)Odisha, India
Dr. Suryakant B Patil - JSPM's ICOER, Wagholi, Pune
Prof. Varadala Sridhar - Vidya Jyothi Institute of Technology, Andhra Pradesh, India
Dr. Inderjeet Kaur - Ajay Kumar Garg Engineering College, Ghiaziabad
Prof. Kuntal Barua - Lakshmi Narain College of Technology, Indore
For Authors
Paper title (14 Bold)- Capital Letter
First Author1, Second Author2 (13 Bold- Times New Roman)
1(Department, College/ University Name, Address, Country Name, Email) (12)
2(Department, College/ University Name, Address, Country Name, Email) (12)
ABSTRACT (12 Bold)
The abstract should summarize the content of the paper. Try to keep the abstract below 350 words. Do not make references nor display equations in the abstract. The journal will be printed from the same-sized copy prepared by you. Your manuscript should be printed on A4 paper (21.0 cm x 29.7 cm). It is imperative that the margins and style described below be adhered to carefully. This will enable us to keep uniformity in the final printed copies of the Journal. Please keep in mind that the manuscript you prepare will be photographed and printed as it is received. Readability of copy is of paramount importance.(12)
Keywords (12 Bold) : About five key words in alphabetical order, separated by comma (12)
I. INTRODUCTION (12 BOLD)
The introduction of the paper should explain the nature of the problem, previous work, purpose, and the contribution of the paper. The contents of each section may be provided to understand easily about the paper. (12)
II. HEADINGS (12 BOLD)
The headings and subheadings, starting with "1. Introduction", appear in upper and lower case letters and should be set in bold and aligned flush left. All headings from the Introduction to Acknowledgements are numbered sequentially using 1, 2, 3, etc. Subheadings are numbered 1.1, 1.2, etc. If a subsection must be further divided, the numbers 1.1.1, 1.1.2, etc.
The font size for heading is 12 points bold face and subsections with 12 points and not bold. Do not underline any of the headings, or add dashes, colons, etc. (12)
III. INDENTATIONS AND EQUATIONS(12 BOLD)
The first paragraph under each heading or subheading should be flush left, and subsequent paragraphs should have a five-space indentation. A colon is inserted before an equation is presented, but there is no punctuation following the equation. All equations are numbered and referred to in the text solely by a number enclosed in a round bracket (i.e., (3) reads as "equation 3"). Ensure that any miscellaneous numbering system you use in your paper cannot be confused with a reference [4] or an equation (3) designation. (12)
IV. FIGURES AND TABLES(12 BOLD)
To ensure a high-quality product, diagrams and lettering MUST be either computerdrafted or drawn using India ink.
Figure captions appear below the figure, are flush left, and are in lower case letters.When referring to a figure in the body of the text, the abbreviation "Fig." is used. Figures should be numbered in the order they appear in the text.
Table captions appear centered above the table in upper and lower case letters. When referring to a table in the text, no abbreviation is used and "Table" is capitalized. (12). Figures and tables should be included in the running text itself
V. CONCLUSION (12 BOLD)
A conclusion section must be included and should indicate clearly the advantages, limitations, and possible applications of the paper. Although a conclusion may review the main points of the paper, do not replicate the abstract as the conclusion. A conclusion might elaborate on the importance of the work or suggest applications and extentions. (12)
VI. ACKNOWLEDGEMENTS(12 Bold)
An acknowledgement section may be presented after the conclusion, if desired.( 12)
REFERENCES (12 BOLD)
This heading is not assigned a number.
A reference list MUST be included using the following information as a guide. Only cited text references are included. Each reference is referred to in the text by a number enclosed in a square bracket (i.e., [3]). References must be numbered and ordered according to where they are first mentioned in the paper, NOT alphabetically.
Examples follow:
ournal Papers:
[1] M Ozaki, Y. Adachi, Y. Iwahori, and N. Ishii, Application of fuzzy theory to writer recognition of Chinese characters, International Journal of Modelling and Simulation, 18(2), 1998, 112-116. (12)
Books:
[2] R.E. Moore, Interval analysis (Englewood Cliffs, NJ: Prentice-Hall, 1966). (12) Note that the title of the book is in lower case letters and italicized. There is no comma following the title. Place of publication and publisher are given.
Chapters in Books:
[3] P.O. Bishop, Neurophysiology of binocular vision, in J.Houseman (Ed.), Handbook of physiology, 4 (New York: Springer-Verlag, 1970) 342-366. (12) Note that the place of publication, publisher, and year of publication are enclosed in brackets. Editor of book is listed before book title.
Theses:
[4] D.S. Chan, Theory and implementation of multidimensional discrete systems for signal processing, doctoral diss., Massachusetts Institute of Technology, Cambridge, MA, 1978. (12)
Proceedings Papers:
[5] W.J. Book, Modelling design and control of flexible manipulator arms: A tutorial review, Proc. 29th IEEE Conf. on Decision and Control, San Francisco, CA, 1990, 500-506 (12)
IAEME Publication Fees
IAEME Publication charges an Article Processing Charge (APC) once an article is accepted for publication. These charges cover the costs of turning a manuscript into a finished article, as well as the costs of hosting, distributing and promoting an article.
Article Processing Charges explained
We are committed to making the costs of publishing as clear as possible and so the charges for each journal are clearly displayed on each journal`s homepage.
When a manuscript is submitted, it passes through the many different departments at Hindawi to ensure the quality checks, peer review, production and promotion of articles is carried out in a timely manner and to a high standard:
✔ The Editorial Screening team, who perform initial technical and ethical checks
✔ The Editorial team, who help the journal`s Academic Editors manage the review process
✔ The Production team, who convert the manuscript to a professionally typeset article and well-structured file format
✔ The Proofing team, who coordinate the proofing process through Hindawi`s Online Proofing System (OPS)
✔ The Editorial Quality Assurance team, who perform a final check to ensure that the manuscript and its review process adhere to the journal`s guidelines and policies
✔ Our Marketing and Communication teams, who ensure your article receives the attention it deserves.
✔ The Technology team, who build and maintain our systems, and develop new systems
The work these teams do contribute to the requirement of an APC and is why the actual amount payable varies depending on the journal in which you wish to publish your article.
IAEME Publication does not request Article Submission Charges, also called Submission Fees. These are due at the time of submission of the manuscript. Nor does IAEME charge per page or for color figures or for any other items for which other publishers are known to charge.
Sources for Article Processing Charges (APC)
Authors of an article are responsible for arranging the payment of APC. Still, that typically does not mean that authors end up paying for the publication of the article. The cost of APC is very often assumed either by the organization funding the research published in the article, or by an Open Access Publishing Fund, or by the institution at which the author is employed.
Fair Dealing - Authors and Publisher
When an author submits a manuscript for review at one of IAEME`s journals, the submitted manuscript should not have been previously published in any form and must not be currently under consideration for publication elsewhere. The manuscript should also not be submitted to any other journal during the review process at IAEME Publication. In case the review process at IAEME takes much longer than stated and the author wishes to end the review process at IAEME, he/she should get in contact with the Editorial Assistant of the journal at IAEME and come to an agreement suitable for both sides after looking at the stage of the manuscript in the review process.
With the APC specified above, the author is given an indication what the APC will be. IAEME is charging Publication Fees (APC) but not Submission Fees. This is to the benefit to authors because payments are only due once the author knows the manuscript is accepted. IAEME with editors and reviewers puts much effort in dealing with the manuscript and expects the authors to stick to the common plan of publishing the paper which is linked to payments of APC as given above at the end of the review process. The author is not legally bound at this time, but it would be seen as a strong breach of academic conduct to back off from the publication process. Once the manuscript is accepted for publication, IAEME will issue an invoice in the Paper Submission and Manuscript Tracking System. This is IAEME `s legal offer and will state the regular APC of the journal. The Corresponding Author is expected to arrange payments of the amount stated at this time. In severe cases IAEME `s Complaints Procedure can be followed and the result of this should be binding to the author.
Refund Policy
Once an article has been accepted for publication, any Article Processing Charges on the article become due. The submitting author accepts responsibility for the Article Processing Charges, and will not issue refunds of any kind except those payments made by mistake or in excess of the amount required.